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EXECUTIVE SUMMARY 



Title I of the No Child Left Behind Act of 2002 (NCLB) is the largest funding vehicle of 
compensatory elementary and secondary education programs for disadvantaged children in the 
United States. This legislation calls on educators to close the gap between low and high 
achievers by using instructional approaches that scientifically based methods have shown to be 
effective. Because of limited knowledge about the effectiveness of most instructional 
approaches, however, it has been difficult for educators to decide how to best use Title I funds to 
improve the educational outcomes of economically disadvantaged students. 

This report lays out the design of a study to evaluate the impact of supplemental reading 
comprehension interventions for students in upper elementary grades of Title I schools. The 
process of finalizing the study’s design — and its focus on reading comprehension 
interventions — occurred over a roughly three-year period, beginning in October 2002. Planning 
for the study took its lead largely from three sources: (1) the Institute of Education Sciences 
(lES); (2) the Title I Independent Review Panel (IRP), which Congress set up to provide the U.S. 
Department of Education (ED) with recommendations on Title I evaluation; and (3) specialized 
technical working groups (TWGs) made up of technical experts in reading comprehension and 
evaluation design. Below, we list the key design parameters that came out of these efforts. The 
report itself provides the rationale for these decisions and discusses design and measurement 
issues, including sample sizes necessary to address the study’s key questions. 



EVALUATION FOCUS 

• The evaluation will focus on testing strategies to improve reading comprehension — 
particularly interventions designed to improve the reading comprehension of students 
in upper elementary grades — so those students can make progress in content areas 
(such as social studies and science) that involve a large amount of expository text. 

• The study is designed primarily so that reliable inferences can be made about the 
effects of each intervention relative to the control group. A secondary focus is on the 
effects of each intervention relative to the other interventions and the effects of the 
interventions on selected groups of students, such as those with limited English 
proficiency (EEP) or low baseline reading skills. 



THE PROCESS FOR SELECTING INTERVENTIONS AND SCHOOLS 

• A three-stage process was used to select interventions. The first stage involved 
soliciting proposals from the field. The second stage involved winnowing down the 
proposals according to a set of initial criteria, which primarily related to whether the 
submissions included requested materials, such as teacher training materials or 
samples of classroom materials. In the third stage, a panel of experts assessed the 
finalists more critically based on factors related to their promise as effective 
interventions. 
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• Schools will be selected such that the evaluation sample is geographically diverse and 
representative of schools with different concentrations of LEP students. In general, 
schools also will have a high concentration of economically disadvantaged students. 



KEY RESEARCH DESIGN PARAMETERS 

• Schools, rather than classrooms or individual students, will be randomly assigned to 
receive either an intervention (treatment) or no intervention (control). 

• Our power analysis indicates that, under reasonable assumptions, a sample of about 
100 schools can achieve the evaluation objectives. These objectives include the 
ability to test four interventions to determine whether they are effective and whether 
some are more effective than others. 

• We recommend spreading the 100 schools across approximately 10 districts. 

• The language in NCLB requires that state assessments be used to gauge student 
performance. To ensure the study can assess the programs’ impacts on students’ 
reading skills, the study will concentrate on assessments that specifically measure 
reading comprehension. 
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I. STUDYING READING COMPREHENSION INTERVENTIONS UNDER TITLE I 



A. STATEMENT OF THE PROBLEM 

Title I of the No Child Left Behind Act of 2002 is the largest funding vehicle of 
compensatory elementary and secondary education programs for disadvantaged children in the 
United States. The legislation calls on educators to close the gap between low and high 
achievers by using instructional approaches that scientifically based methods have shown to be 
effective. Because knowledge about the effectiveness of most instructional approaches is 
limited, however, it has been difficult for state and local educators to decide how to best use Title 
I funds to improve the educational outcomes of economically disadvantaged students. 

This report lays out the design of a study to evaluate the impact of supplemental reading 
comprehension interventions for students in upper elementary grades of Title I schools. The 
process of finalizing the study’s design — and its focus on reading comprehension interventions 
and their effects on student comprehension of expository text (both in general and in science and 
social studies) — occurred over a roughly three-year period. It began in October 2002, when the 
Institute of Education Sciences (lES) contracted with Mathematica Policy Research, Inc. (MPR) 
to help identify issues relevant to the evaluation of Title I and to propose feasible evaluation 
design strategies. The process of finalizing design decisions continued after lES contracted with 
MPR to conduct the evaluation in October 2004. 

National statistics on reading achievement provide more insight into the decision to focus 
the study on reading. The most recent (2005) National Assessment of Educational Progress 
indicates that 36 percent of the nation’s fourth-grade children have difficulty reading (U.S. 
Department of Education [ED] 2006). Other estimates suggest that as many as 30 percent of 
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elementary, middle, and high school students have reading problems that severely curtail their 
educational progress and ultimate educational attainment (Moats 1999). 

Educators and policymakers seeking to improve this situation, however, must wade through 
a massive body of literature to assess what is known and not known about children’s reading. 
Although the National Reading Panel Report (2000), for example, provides some guidance for 
educators and policymakers, it does not tell us what interventions are best to teach students to 
comprehend the information in the difficult textbooks they will encounter in school and to learn 
from that information. This is a particularly vexing problem for children from low-income 
households, because these children may well be below grade level in their reading even as they 
enter the intermediate grades. Thus, in establishing best practices for reading in connection with 
subject matter learning, it is crucial to identify and test intervention programs designed to help 
students comprehend and learn content from expository texts. 

Planning the design for the study took its lead largely from three sources: (1) lES; (2) the 
Title I Independent Review Panel (IRP), which Congress set up to provide ED with 
recommendations on Title I evaluation; and (3) specialized technical working groups (TWGs) 
made up of technical experts in reading comprehension and evaluation design.^ The decision to 
focus on reading comprehension, specific content areas, and students in upper elementary grades 
reflects discussions of the three groups noted above. It also reflects the fact that (1) lES was 
already devoting considerable effort to understanding the effectiveness of the Reading Eirst 
program, which targets younger children; and (2) even if the approaches funded under Reading 
Eirst are effective, many disadvantaged children may still be struggling readers as they enter the 



'Appendix A of Glazerman and Myers (2004) lists the members of the original design task TWG. Appendix A 
of this report lists the members of the TWGs for the current evaluation. 
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higher elementary grades. Additional details of the original design effort are provided in 
Glazerman and Myers (2004) (hereafter, referred to as the original design report). 

B. CONCEPTUAL FRAMEWORK 

This study provides a unique opportunity to address questions critical to understanding the 
effectiveness of reading comprehension interventions. We formulated a conceptual framework 
for the study to help organize our approach to selecting and implementing interventions, setting 
up the experimental design, collecting data, and computing intervention impacts. The 
framework (Figure I.l) illustrates what we believe are the major concepts that are relevant for the 
evaluation of reading comprehension interventions and the pathways through which students’ 
general reading comprehension and comprehension of expository text in content areas, such as 
science and social studies, will be influenced during the evaluation period. The framework 
suggests that the reading comprehension curriculum and instruction as it is designed (Planned 
Instruction and Curriculum box) indirectly affects student outcomes because its effect is 
mediated by how teachers implement it. For example, whether they use the instructional 
materials provided by the curriculum developers or adapt other materials for their classes will 
potentially mediate the effect of the intervention. 

Furthermore, we hypothesize that the school and classroom context, as well as teacher 
characteristics, will affect how the reading curricula and instruction are implemented. For 
example, as teachers implement the curricula, the level of support they receive from school 
principals may affect that implementation. 



^ We have not distinguished the curriculum and instruction being offered as part of the reading comprehension 
interventions from the approaches that would normally be used in the absence of the interventions. The distinction 
is made later, when the experimental design is set up and schools are randomly assigned to interventions or a control 
group (nonintervention instruction). 
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FIGURE 1. 1 



A CONCEPTUAL ERAMEWORK EOR MEASURING THE EEEECTIVENESS 
OE READING COMPREHENSION INTERVENTIONS 




The quality of teacher professional development may also affect the implementation of the 
curricula. For example, impacts of the curricula may be greater for students whose teachers have 
participated in larger amounts of professional development. The framework also suggests that 
students’ initial reading skills will affect their outcomes. For example, students who enter the 
experiment with stronger initial reading skills in either fluency or comprehension (both of which 
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will be measured at baseline) may develop comprehension skills at different rates than those with 



weaker skills in either domain. 



C. RELEVANT RESEARCH 

Research has not helped answer the question of how best to teach students to comprehend 
the information in the textbooks they will encounter in school. Students typically develop their 
reading skills in the early elementary grades using narrative texts. When students enter the upper 
elementary grades, they need to develop ways to understand informational (expository) text. 
Making this transition can be difficult for some students, particularly those who have not fully 
developed their reading skills. The text below describes what the existing research can tell us 
about reading. 

The National Reading Panel (NRP) (2000) identified five areas of reading in which research 
has been conducted to allow for some instructional recommendations for teachers. These areas 
of reading include (Armbruster and Osborn 2001): 



• Phonemic Awareness. Phonemic awareness is the ability to think about and 
notice the individual sounds in spoken words; for example, the word “cat” is 
made up of the sounds Id Id /t/. Phonemic awareness involves understanding 
that spoken words are made up of individual speech sounds. 

• Phonics. Phonics is the understanding of the relationship between spoken 
sounds and written letters. To be able to read words, children must understand 
this relationship. 

• Fluency. Fluency is the ability to read a text effortlessly and with expression. 
Fluency consists of accurate reading, at a reasonable rate and with appropriate 
expression. 



^ The RAND Reading Study Group (2000) also mentioned several other critical elements of comprehension 
instruction, such as fluency and the need for specific instruction within the context of subject matter learning (for 
example, in social studies or science). 
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• Vocabulary. Vocabulary refers to word knowledge. Children must know and 
understand many words to be able to comprehend what they hear and read. 
Greater vocabulary knowledge leads to increased comprehension. 

• Comprehension. Comprehension is making meaning from text. The goal of all 
reading is understanding what is read. 

The NRP (2000) review of the research on comprehension identified specific comprehension 
strategies, or procedures and routines, that have been demonstrated to be effective for improving 
comprehension."^ However, little is known about the role of these strategies in the overall 
comprehension curriculum or how these strategies may help students learn subject matter content 
(RAND Reading Study Group 2000). 

While little is known about the extent to which comprehension strategies can improve 
content area learning, research does indicate that all readers find expository text more difficult to 
comprehend than narrative text (Graesser et al. 1991). There are two main reasons why readers 
have difficulty with expository text: (1) the organization of expository text is often unfamiliar, 
and (2) the content of expository text is often unfamiliar. The combination of an unfa mi liar 
organization and unfamiliar content makes information in expository texts, especially textbooks, 
difficult to comprehend. In fact, researchers have hypothesized that the legendary “fourth-grade 
slump” may well be due to students’ inexperience with expository texts (Chall et al. 1990). 

Below, we review the recent research to illustrate which reading comprehension strategies 
may be effective. This information is critical in designing an evaluation, which includes the 
selection of interventions that could affect children’s ability to extract meaning and 
understanding from text. Toward this end, we present the following: 



Some of the strategies are learner strategies, such as using prior knowledge, generating questions, and 
summarizing, that readers use to make sense of text. Other strategies are teaching strategies, such as cooperative 
learning and graphic organizers, that teachers use in the classroom to help students make sense of text. 
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• Key reading comprehension theories and concepts 

• Findings from several syntheses that present information on the effectiveness of 
reading comprehension strategies 

• An assessment of the limitations of the current research on reading comprehension 

1. Reading Comprehension Theories and Concepts 

Research on comprehension is based on a few theories and concepts: 



• Schema Theory. This theory suggests that what students know about a topic or a 
construct influences how much they can or will learn by reading a passage that 
addresses that topic (Anderson and Pearson 1984). Schema theory, which was the 
building block for reading comprehension research beginning in the 1970s, implies 
that the more students read and learn about a topic, the easier it will be for them to 
understand that topic the next time it comes up. 

• Metacognition. Metacognition refers to students’ conscious awareness of the 
cognitive processes they use and anything related to those processes (Flavel 1976). In 
reading, this concept explains students’ awareness of whether they understand what 
they read and their ability to change and modify the strategies they use in order to 
help them comprehend better. 

• Vygotsky’s Theory of Cognitive and Early Social Constructivism. This theory 
suggests that teachers serve as models and facilitators of verbal interactions that lead 
to internal understandings about comprehension processes (1964, cited in Palincsar 
1986). Palincsar and Brown (1984) built upon this theory by postulating that students 
would become more adept at using comprehension strategies through the use of 
shared group dialogue among small groups of readers who jointly build meaning from 
text. 



Each of these theories led to different lines of research on comprehension instruction.^ 
Eventually, a large body of work was generated, and researchers began to ponder what the 
collective evidence had to say and whether broader lessons could be drawn. To try to answer 
these questions, researchers generated syntheses of the existing research. Next, we present the 



^ For example, research based on schema theory often involved testing whether an intervention to build 
students’ knowledge on a given topic would help students’ reading comprehension (see, for example, Au and 
Crowell, 1979 and Langer 1981 and 1984). Research based on Vygotsky’s theory involved testing whether the use 
of focused, group dialogue would help students make sense of text. 
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findings from these syntheses and related research that support the reading comprehension 
interventions included in the study. 

2. Review of Research Syntheses 

The review of research syntheses suggests six techniques and approaches for teaching 
reading comprehension: (1) teach strategies; (2) use proven methods for instructional delivery; 
(3) use embedded instruction; (4) teach with highly engaging, interesting texts; (5) use 
cooperative learning; and (6) participate in ongoing professional development. In discussing 
these techniques and approaches, we address some unresolved issues in the comprehension 
instruction research that may be informed by this evaluation. We end by mentioning some 
limitations in the research. 

Teach Strategies. One approach linked to teaching reading comprehension effectively is to 
teach students to use comprehension strategies to help them actively make meaning out of the 
texts they read. Research has established that teaching students to use these strategies helps them 
better understand the text they read. Two recent reviews of the literature — the NRP report and a 
review of intervention research for students with reading disabilities (Gersten et al. 2001) — 
indicate that, in general, interventions that teach flexible use of multiple strategies to improve 
comprehension produce large improvements in reading comprehension. 

There is also general agreement in the literature on the types of strategies that should be 
included in strategy instruction (Pearson et al. 1992; Pressley 2002; National Reading Panel 
2000; RAND Reading Study Group 2000; Pressley et al. 1989). These strategies generally fall 
into one of three groups: (1) summary and prediction, (2) question generation, and (3) 
understanding of text structure and use of graphic organizers. We discuss each of these groups 
next: 
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1 . Summary and Prediction. In summarizing, students condense information they have 
read into the essential or main points. Summarizing encompasses multiple strategies 
for comprehending text, such as determining what is important, categorizing, and 
organizing information (Brown and Day 1983). The NRP report found that 
comprehension of expository text is enhanced when students are taught to record 
their ideas about what they read and to summarize these ideas after reading long 
passages. In prediction, students make predictions about what they are about to read 
based on subtitles or material in the preceding paragraphs. For example, before 
reading a passage, students make predictions about what they are about to read and, 
after reading the passage, they evaluate the accuracy of their predictions. Hansen 
(1981) and Hansen and Pearson (1983) have shown that teaching students to use 
prediction while reading improves their comprehension. 

2. Question Generation. Question generation involves asking questions while reading 
and attempting to answer those questions. This strategy makes readers more active 
in the comprehension process and focuses the readers’ attention, particularly on the 
information that will answer the self- generated questions. This makes the text 
content easier to understand and to remember (Martin and Pressley 1991). Question 
generation seems especially useful for learning material from expository text 
(Pressley et al. 1992; Wood et al. 1990; and Rosenshine et al. 1996). The NRP 
report also found support for interventions that provided opportunities for students to 
ask and answer their own questions about the text. 

3. Understanding of Text Structure and Use of Graphic Organizers. Text structure 
refers to how writers and readers organize important information in a text (Meyer et 
al. 1980). Four basic text structures in expository text are (1) compare-contrast, (2) 
cause-effect, (3) explanation, and (4) sequencing.^ Readers who are aware that 
authors typically use these text structures to organize information tend to recognize 
and retain the important information more easily (Snow 2002). There have been few 
studies on the use of text structure to understand expository text because many of the 
expository texts students read in school have a “mixed” structure (for example, some 
cause-effect with a good deal of explanation or sequence). The text structure studies 
that do exist most often involve the use of graphic organizers (visual presentations 
of information to help students understand text). The NRP report suggests that text 
structures and graphic organizers may be promising strategies for understanding 
expository text — however, because there are few studies on this issue, it does so 
tentatively. 



® A compare-contrast structure requires readers to compare and contrast one event or object with another. For 
example, tornadoes and hurricanes are alike in that they both can cause damage to the environment, but they are 
different in that they arise from different weather patterns. A cause-effect structure requires readers to infer one 
event causing another (for example, hurricanes are caused by . . .). An explanation structure provides an explanation 
for something (for example, there are many different kinds of precipitation. One kind of precipitation is...). A 
sequencing structure lays out a linear sequence of events (for example, first, second, next, finally). 
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Use Proven Methods for Instructional Delivery. A second approach linked to teaching 
reading comprehension effectively is to teach strategies using proven methods of instructional 
delivery. Here, we outline three recommended methods of delivering instruction: 



1. Direct, or Explicit, Instruction. Direct, or explicit, instruction of comprehension 
strategies involves modeling how the comprehension strategy or skill is used, 
guiding practice with feedback from a teacher, and providing opportunities for 
students to independently practice using the strategy or skill on various reading 
materials (Rosenshine and Stevens 1986). Early randomized trials using explicit 
instruction and targeting only one specific skill or strategy demonstrated significant 
improvements in students’ understanding of expository text, ability to critically 
analyze arguments, and retention of content (Adams et al. 1982; Darch and Gersten 
1986; Darch and Kame’enui 1987; Lloyd et al. 1980; Patching et al. 1983). 

2. Direct Explanation of Strategies. Direct explanation of strategies is similar to 
explicit instruction in that the three most important elements of explicit instruction — 
modeling, guided practice, and feedback — are included, however, there is more 
focus on the methods used by teachers to explain the strategies. Teachers first name 
and explain, or define, the particular strategy for the students, describe when and 
how it might best be used, and tell students why the strategy is important for 
improving reading. They next engage in a significant amount of explanation and 
cognitive modeling to show the students how to use the strategy. The teacher 
engages the students in practicing the strategy in teacher-mediated activities until 
students are able to use the strategy independently. The direct explanation of 
strategies has been shown to be particularly effective in comprehension strategy 
instruction (Duffy et al. 1987; Duke and Pearson 2002; National Reading Panel 
2000; RAND Reading Study Group 2000). 

3. Differentiated Instruction. Differentiated instruction is instruction that is 
individualized in an attempt to meet the specific needs of each student (Fuchs and 
Fuchs 2005). Students differ in their instructional needs, not only in reading level, 
but also in the strategies and skills they need to be taught. Interventions that attempt 
to target individual student needs, rather than group needs, should result in improved 
comprehension. For example, computer programs can differentiate instruction by 
identifying particular skills in which students are weak and providing targeted 
practice in those .skills. 



Use Embedded Instruction. A third approach linked to teaching reading comprehension 
effectively is to embed strategy instruction into the reading of text in different academic content 
areas. The idea behind this approach is that if students are taught the strategies using the 
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expository texts they read during their content area classes, they will more likely transfer their 

n 

use of the strategies to texts they read in other content areas, and on their own. 

Interventions that embed strategy instruction into content area reading activities still include 
the direct explanations and teacher cognitive modeling associated with the direct explanation 
approach to teaching comprehension strategies. However, these periods of explicit instruction are 
generally shorter, individual strategies may be introduced more closely together, and the 
occasion to teach a strategy is often more natural, arising from a spontaneous classroom event or 
interaction. 

The most well known of the embedded strategy instruction interventions is transactional 
strategies instruction (Pressley et al. 1998; Pressley 2002). In this approach, the teacher and 
students jointly develop meaning by strategic reasoning through a text together. Studies suggest 
this approach may be highly effective for improving reading comprehension (Brown et al. 1996; 
Collins 1991; Anderson 1992; Anderson and Roit 1993). 

Teach with Highly Engaging, Interesting Texts. A fourth approach linked to teaching 
reading comprehension effectively is to use highly engaging, interesting texts when teaching 
reading comprehension. Interesting texts are defined as texts that have vivid details, are relevant 
to the task, are easily accessible to students, and have colorful photographs and illustrations 
(Schraw et al. 1995). Many studies have demonstrated the value of using interesting texts to 
teach reading comprehension (Anderson et al. 1987; Asher 1980; Guthrie et al. 1998; Guthrie et 
al. 2000a; Guthrie et al. 2000b). 

^ Some researchers argue that, when strategy instruction is taught in isolation (for example, on workbook pages 
during reading instruction), students do not make the transfer from workbook pages to their own reading of 
expository texts, especially to what they read for social studies or science (Pearson and Fielding 1991; Pressley 
2000). This may occur because students perceive the learning activity to be completing the workbook pages, not 
learning strategies to use with real texts. 
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Use Cooperative Learning. A fifth approach linked to teaching reading comprehension 
effectively is to use cooperative learning strategies in the classroom. In cooperative learning, 
students can interact with peers when discussing text. Research has shown that interactions 
between peers are critical in getting students to express their thoughts and, ultimately, to 
internalize strategies. Both the NRP report and Gersten et al. (2001) noted that giving children 
the chance to practice a strategy in a small group contributed to the success of most 
interventions. 

Participate in Ongoing Professional Development. A sixth approach linked to teaching 
reading comprehension effectively is to provide teachers with ongoing professional development 
in the teaching of reading comprehension strategies. According to the NRP report, ongoing 
professional development is important for successful implementation of instructional plans 

o 

related to reading comprehension strategies. Several studies demonstrate that, when teachers 
are provided with sufficient professional development, their teaching of comprehension 
strategies improves (Anderson 1992; Brown et al. 1996). In addition, Duffy et al. (1987) found 
that ongoing professional development consisting of one-on-one coaching, collaborative sharing, 
and lesson observation and feedback helped teachers learn how to teach comprehension 
strategies. 

3. Limitations in the Research 

Many studies in the syntheses and related research on comprehension instruction have one 
or more serious methodological flaws that limit their scientific merit and, therefore, their value in 
program and policy development. The most consistent flaws include the use of (1) quasi- 

* The report concluded that teachers can learn to use effective comprehension instruction practices in the 
classroom. 
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experimental designs to estimate impacts, (2) materials the researchers developed themselves 
that were sometimes quite different from those that students typically encountered, (3) outcomes 
that were closely aligned to the strategy being taught and thus could not be used to assess general 
comprehension ability, and (4) short-term follow-up data on outcomes. In addition, many studies 
did not assess fidelity of implementation — the extent to which the intervention as implemented 
resembled the intervention model. Finally, other than Fuchs et al. (1997) and Vaughn et al. 
(2001), the majority of studies were based on instruction delivered to students by well-trained 
graduate students or teachers personally trained by the researchers. Thus, we know little about 
how useful the interventions would be in classrooms in which teachers were not exposed to such 
training. Because of these limitations, one should interpret many of the findings as suggestive of 
the effects we might expect to obtain in the context of a well-designed experiment with random 
assignment of subjects to intervention or control groups. For these reasons, we are limited in 
what we can infer from this body of research. 

D. DESIGN OVERVIEW 

The experimental design for the study reflects (1) careful consideration of the questions 
posed by lES, the Title I IRP, and TWGs that worked with MPR; (2) our knowledge that the 
interventions would be implemented in classrooms; (3) IBS’s suggestion that up to four 
interventions should be considered; (4) the study team’s suggestion that the focus of the study 
would be upper elementary grades (fifth graders); and (5) concerns about spillover effects. To 
address these issues, we will use a cluster randomized experiment with four intervention groups 
and a control condition. This straightforward design is based on the random assignment of 
schools to the intervention groups or a control group. 

Because the study involves estimating impacts of reading comprehension interventions 
relative to what students would have experienced without the intervention, we believe it is 



13 




important that all aspects of the schools other than the intervention, including the allocation of 
students to classrooms by school principals, remain the same as they were before the 
intervention. Accordingly, we will suggest that school principals make classroom assignments 
as they normally would. Therefore, we recommend that students from all fifth-grade classrooms 
be included in the interventions and evaluation. 

The study includes baseline data collection during fall 2006 and follow-up data collection 
during spring 2007, with the potential for a second year of data collection, depending on the 
study’s first-year results. Thus, the implementation of the interventions (and the measurement of 
the impacts of those interventions) will cover roughly one school year. This data collection 
schedule implies that these curricula have the potential to affect outcomes during a single school 
year. 

When estimating sample size requirements for the design, we factored in the following four 
statistical considerations: (1) the need to account for multiple comparisons when conducting tests 
of statistical significance, (2) whether we can estimate district fixed effects in the impact 
analysis, (3) the benefits of a baseline test to increase precision, and (4) IBS’s desire to detect 
differences between each intervention group and the control group equal to an effect size of 0.25 
or larger with a high probability. Furthermore, we worked from key assumptions concerning the 
expected number of fifth-grade classes within schools, the number of students within classes, and 
the expected correlation between pre- and post-test measures of reading achievement.^ 



^ To estimate the number of schools needed for the evaluation, we assumed the typical elementary school 
would have three fifth-grade classrooms and about 26 students in each classroom. We also assumed that (1) 50 
percent of the variance in test scores could be accounted for with knowledge of the pretest scores, (2) the desired 
probability of detecting a significant impact if there was an impact was 0.80, and (3) the probability of rejecting the 
hypothesis of no impact was 0.05 (two-sided). Furthermore, we assumed that 10 percent of the total variance in test 
scores was attributable to between-school differences. 
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The statistical considerations outlined above suggest an experimental design that includes 
100 schools (20 in each of four intervention groups and 20 in the control group). With a total 
sample of 100 schools, it is important to include multiple school districts in the evaluation. In 
addition, it will be valuable both to have the districts geographically dispersed and to minimize 
the burden on the districts by not selecting too many schools within any one district. Given these 
considerations, we are planning on a design with roughly 10 school districts, with each district 
contributing about 10 schools to the study. A natural representation of an experimental design in 
which schools are randomly assigned within districts to intervention and control groups is the 
randomized blocks design (Kirk 1968). Blocking on school districts not only ensures a 
representation of schools in districts and interventions but also could increase the precision with 
which impacts can be estimated. 

In addition to using the school district as a blocking factor in the experimental design, we 
will consider using additional blocking factors when implementing the design. The blocking 
factors we will consider include the percentage of students within schools eligible for free and 
reduced-price lunches and average reading achievement for fifth graders. As the recruiting of 
districts and schools progresses, we will assess the need for blocking on these factors, depending 
on the extent to which these characteristics vary across the participating schools in each district. 

E. PLAN OF THE REPORT 

Chapter II outlines the procedures for selecting interventions and schools for the study. 
Chapter III discusses the random assignment design, sample size requirements, and data 
collection and data analysis plans. 
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II. SELECTING INTERVENTIONS AND SCHOOLS 



A. INTERVENTION SELECTION AND IMPLEMENTATION 
I. Intervention Selection 

To select the best mix of reading comprehension programs, we designed a competitive 
process that drew on the expert judgments of a panel of nationally recognized reading 
researchers. The competition was announced in Education Week, and potential candidates were 
notified by email. Interested parties could contact the study team and request that a hard copy of 
the Request for Proposals (RFP) be mailed to them. Shortly afterward, we posted the RFP on the 
study’s website. The RFP invited researchers and partnerships of researchers, publishers, and 
organizations involved in professional development for instruction in reading comprehension to 
apply to participate in the study by submitting a proposal within the next 30 days. 

Seven candidates submitted a proposal in response to the January 12, 2005, web posting of 
the RFP. We then began a three-stage review process. First, we reviewed all proposals for the 
following minimum qualifications: 

• Submission includes a technical proposal. 

• Submission includes samples of teacher training materials. 

• Submission includes samples of classroom materials (including classroom support 
materials). 

• Submission includes a budget. 

Second, we forwarded the proposals that met the minimum qualifications (all seven did so) 
to the panel of experts, who reviewed and evaluated the proposals using the criteria in the table 
below: 
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Criteria 



Points 



I. Summary description of intervention, theoretical and empirical support for the intervention 35 points 
content, and evidence of the intervention’s efficacy or effectiveness 

a. Underlying theory with empirical support (25 points) 

b. Evidence to support the effectiveness of the intervention (10 points) 

30 points 

II. Quality of the proposed intervention design 

a. Objectives of intervention, including description of teacher practices and skills that 
comprise the intervention (10 points) 

b. Intensity and quality of teacher training design and follow-up support design (10 points) 

c. Quality of training and support materials, quality of classroom activity materials, and 
quality of any intervention-specific assessments (10 points) 

20 points 

III. Institutional capability to provide training and follow-up support 
(staff qualifications, capacity to schedule and manage training) 

15 points 

IV. Appropriateness of intervention 

a. For target population (grade 5, Title I schools) (5 points) 

b. For content (comprehension of expository text in social studies or science) (10 points) 



We also considered the reasonableness of proposed costs for program materials, training, 
and follow-up support, in conjunction with the proposed training design and the justification 
provided for costs. 

Third, the five bidders with the highest ratings from the reading experts were selected for in- 
person interviews on March 10-11, 2005. The finalists presented an overview of their proposals 
and responded to questions from the expert panel, which then recommended three programs to 
participate in the pilot study. (The next section contains more information about the pilot study.) 
IBS approved these recommendations. 

After completing this competition, we decided to hold a second competition, for two 
reasons: (1) the evaluation plans called for up to four reading comprehension programs and (2) 
we believed that a second solicitation would bring forward additional promising programs. The 
second competition followed procedures identical to those used in the first competition. We 
reissued the RFP on April 29, 2005, and another six proposals were submitted. The expert panel 
recommended three program developers for interviews, which were conducted on June 30, 2005. 
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Based on the interviews, the expert panel and project staff selected two additional interventions 
for the pilot study. As with the first competition, IBS approved the recommendations. 

2. Intervention Implementation 

Each of the five program developers selected for the pilot identified three or more schools in 
which it would implement its intervention in the 2005-2006 academic year. This pilot allowed 
developers to augment and refine their teacher training design and materials, as well as their 
classroom materials, and to implement their interventions in a small number of classrooms (nine 
classrooms per intervention) under conditions consistent with the upcoming full evaluation. The 
reading experts reviewed all implementation plans, designs, and materials before they were 
piloted. The study team observed (1) developers’ training of teachers, and (2) teachers’ 
classroom instruction using the developers’ curricula. They then provided feedback to the 
developers to guide further refinements and improvements in their designs and materials before 
the full implementation year. Developers were required to respond to the comments but could 
decide what revisions, if any, they wanted to make. 

In the spring of the 2006-2007 school year, we will select up to four of the five 
interventions that constitute the best mix of programs to be continued after the pilot-test year. 
The selection of interventions for the full implementation will be based on the criteria in Table 
II. 1 and will follow these steps: 

1. We will provide the Intervention TWG with (I) expert reviews of developer materials, 
(2) developer responses to the reviews, (3) reports on teacher training observations and 
the first round of classroom observations, and (4) a summary of the TWG’s evaluation of 
the developers’ initial proposals. 

2. Each Intervention TWG member will independently review the above materials and 
assess each intervention based on the criteria provided (Table II. 1). Each TWG member 
will then provide a preliminary assessment of the curricula to the study team. 
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3. We will convene a teleconference of the TWG to discuss the interventions and identify 
key strengths and weaknesses and distinguishing features of each intervention. 

4. We will send lES recommendations of which reading programs should continue into the 
full implementation. 

TABLE II. 1 

SELECTION CRITERIA EOR EULL IMPLEMENTATION 

1 . Meets contractual requirements for pilot-test year. 

2. Ease of use for teacher. 

a. Materials and activities are readily integrated into classroom routines (e.g., teacher’s guide provides lesson 
plans that are easy to follow; student materials have a wraparound teacher’s guide; activities, including 
computer applications, are functional). 

b. Teacher-friendly materials (e.g., lessons follow similar format; use of color or graphics makes lesson plans 
or scripts appealing and easy to follow). 

3. Intensity/duration of teacher professional development. 

a. Duration of initial training and follow-up support are commensurate with (or adequate for) program 
complexity. 

b. Initial training and follow-up support are sufficient in motivating teachers to implement program as intended. 

c. Initial training and follow-up support are well specified. 

4. Program is well specified and robust. 

a. Program activities are clearly outlined and tied to expository reading comprehension objectives. 

b. Program activities can be satisfactorily implemented by teachers with a range of teaching skill or experience. 

5. Developer has the capacity to support large-scale implementation. 

a. Developer has sufficient staff to support up to 20 schools. 

b. Training and support model is adequate to ensure fidelity of implementation. 

6. Theoretical and empirical support for the program’s content and effectiveness. 

a. Effectiveness of program’s strategies based on prior theory or research. 

b. Effectiveness of program based on program-specific empirical research. 



The full implementation of up to four selected interventions in 100 schools will occur in the 
2006-2007 school year. During this year, the intervention team will continue to monitor the 
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implementation of each intervention but will not provide formative feedback on designs or 
materials. Chapter III describes how the information the study team collects about the 
implementation of the curricula will be used. 

B. SELECTING AND RECRUITING DISTRICTS AND SCHOOLS 

Recruiting districts and schools is one of the most formidable challenges the study faces 
because (1) the schools for the evaluation must meet specific criteria and (2) there is no federal 
requirement to participate in the evaluation. Our recruiting approach deals with this challenge by 

(1) initially targeting more school districts and schools than will actually end up in the study, (2) 
seeking support from high-level education officials, (3) appealing directly to school staff who 
may be affected, and (4) offering modest cash payments to defray the potential burden associated 
with participation. 

I. Identifying Eligible Districts and Schools 

The criteria we used for selecting eligible districts reflect three main concerns: (1) the 
recommendation of the Title I national assessment IRP to focus on high-poverty Title I schools, 

(2) the need to include districts with enough schools to make it possible for the study to assign 
two schools to each of the treatment groups (four experimental and one nonexperimental), and 

(3) the desire to select schools and districts that would provide face validity to the study. 

We first created a list of eligible districts. To create the list, we used data from the 2003 
Common Core of Data (CCD) school- and district-level files. To be included, each district had 
to have: 



• At least 12 schoolwide Title I schools' 



* Individual public schools with poverty rates above 40 percent may use Title I funds, along with other federal, 
state, and local funds, to operate a schoolwide program to upgrade the instructional program for the whole school. 
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• At least 40 percent of students in the schools noted above who were eligible for 
free or reduced-price lunch 

• At least 60 fifth-grade students in the schools noted above 

A total of 157 school districts met the criteria listed above? 

To begin our recruiting efforts, we selected a subset of 28 districts from the pool of 157 
eligible districts. The process used to identify the 28 districts, which we will approach first about 
participating in the study, reflects the desire to include districts that are geographically dispersed 
throughout the United States. We randomly selected 7 districts from each of four regions (south, 
west, midwest, and northeast) for a total of 28 randomly selected districts. Table II.2 lists 
the randomly selected districts. 

Our goal was not to create a statistically representative sample of districts. Instead, we used 
the stratified random selection to help us select a diverse set of districts to approach about 
participating in the study. While the set of randomly selected districts is fairly well balanced on 
the key characteristics shown in the table, there were a few areas in which the sample could have 
been more balanced. In particular: 



^ Schools that participated in the pilot year cannot participate in the full implementation of the study, because 
teachers in those schools would have a year of experience with the reading programs and would thus not be 
comparable to teachers at nonpilot schools (either experimental or nonexperimental) selected for the study. 

^ The south region contains Alabama, Arkansas, the District of Columbia, Delaware, Georgia, Florida, 
Kentucky, Louisiana, Maryland, Mississippi, North Carolina, Oklahoma, South Carolina, Tennessee, Texas, 
Virginia, and West Virginia. 

The west region contains Arizona, Alaska, California, Colorado, Idaho, Montana, Nevada, New Mexico, 
Oregon, Utah, Washington, and Wyoming. 

^ The midwest region contains Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, 
North Dakota, Ohio, South Dakota, and Wisconsin. 

® The northeast region contains Connecticut, Maine, Massachusetts, New Hampshire, New Jersey, New York, 
Pennsylvania, Rhode Island, and Vermont. 



22 




• The set of districts would benefit from more representation among suburban and 
mral schools. Among the 157 eligible districts, there were seven suburban (and one 
mral) schools on average, while the randomly selected set of districts included just 
three suburban (and no rural) schools on average.’ 

• The set of districts would benefit from more geographic diversity in the south and 
west regions (where many Texas and California districts were selected). 



TABLE II.2 

LIST OF RANDOMLY SELECTED DISTRICTS 



District Name 


City 


State 


Number Urban 
Schoolwide Title 
I Schools with at 
Least 60 Fifth 
Graders 


Number 
Suburban 
Schoolwide Title 
I Schools with at 
Least 60 Fifth 
Graders 


Number Rural 
Schoolwide Title 
I Schools with at 
Least 60 Fifth 
Graders 


Midwest 












City of Chicago School District 


Chicago 


IL 


196 


1 


0 


Cincinnati City School District 


Cincinnati 


OH 


17 


2 


0 


Cleveland Municipal City Schools 


Cleveland 


OH 


54 


0 


0 


Columbus Public Schools 


Columbus 


OH 


41 


2 


0 


Milwaukee School District 


Milwaukee 


WI 


45 


0 


0 


Gary Community Schools 


Gary 


IN 


15 


0 


0 


Omaha Public Schools 


Omaha 


NE 


18 


1 


0 


Northeast 












Philadelphia City School District 


Philadelphia 


PA 


104 


0 


0 


Jersey City Public Schools 


Jersey City 


NJ 


0 


17 


0 


Bridgeport School District 


Bridgeport 


CT 


16 


0 


0 


Rochester City Schools 


Rochester 


NY 


20 


0 


0 


Springfield Public Schools 


Springfield 


MA 


15 


0 


0 


Providence School District 


Providence 


RI 


21 


0 


0 


Brockton 


Brockton 


MA 


12 


0 


0 


South 












Goose Creek School District 


Baytown 


TX 


12 


0 


0 


Richardson Independent School District 


Richardson 


TX 


15 


0 


0 


Fort Worth Independent School District 


Fort Worth 


TX 


41 


2 


0 


McAllen Independent School District 


McAllen 


TX 


13 


0 


0 


Birmingham City 


Birmingham 


AL 


22 


0 


0 


Ector County Independent School District 


Odessa 


TX 


20 


0 


0 


Fulton County 


Atlanta 


GA 


1 


14 


3 


West 












Sacramento City Unified 


Sacramento 


CA 


18 


5 


0 


Compton Unified 


Compton 


CA 


15 


4 


0 


Granite School District 


Salt Lake City 


UT 


0 


13 


0 


Oxnard Elementary 


Oxnard 


CA 


13 


0 


0 


Alhambra City Elementary 


Alhambra 


CA 


4 


8 


0 


Adams-Arapahoe (Aurora) 


Aurora 


CO 


14 


0 


0 


Albuquerque Public Schools 


Albuquerque 


NM 


24 


8 


0 



^ The urban, suburban, and rural groups were created using location information from the CCD. Large city 
(population greater than 250,000) and mid-size city (population less than 250,000) categories are included in the 
urban group. Fringe of large city, fringe of mid-size city, and large town (population greater than 25,000) are 
included in the suburban group. The small town (population less than 25,000) and rural categories are included in 
the rural group. 
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To address these issues, we recommended that lES make the following changes to the 



districts randomly selected: 



• Replace Philadelphia, Pennsylvania, with Paterson, New Jersey, because Paterson 
would provide suburban schools. 

• Replace Goose Creek, Texas, with Marion County, Florida, because Marion County 
would provide suburban and rural schools and more geographic diversity in the 
southern region. 

• Replace Ft. Worth, Texas, with Robeson County, North Carolina, because Robeson 
County would provide rural schools and give us more geographic diversity in the 
southern region. In addition, the list already includes another distriet (Riehardson, 
Texas) in the Dallas area. 

• Replaee Compton, California, with Taeoma, Washington, beeause Taeoma would 
provide more geographie diversity in the west region. In addition, the list already 
ineludes two other districts (Oxnard and Alhambra) in the Fos Angeles area. 



Table II.3 reflects these recommendations. 



TABLE II.3 

LIST OF RECOMMENDED DISTRICTS 



District Name 


City 


State 


Number Urban 
Schoolwide Title 
I Schools with at 
Least 60 Fifth 
Graders 


Number 
Suburban 
Schoolwide Title 
I Schools with at 
Least 60 Fifth 
Graders 


Number Rural 
Schoolwide Title 
I Schools with at 
Least 60 Fifth 
Graders 


Midwest 












City of Chicago School District 


Chicago 


IL 


196 


1 


0 


Cincinnati City School District 


Cincinnati 


OH 


17 


2 


0 


Cleveland Municipal City Schools 


Cleveland 


OH 


54 


0 


0 


Columbus Public Schools 


Columbus 


OH 


41 


2 


0 


Milwaukee School District 


Milwaukee 


WI 


45 


0 


0 


Gary Community Schools 


Gary 


IN 


15 


0 


0 


Omaha Public Schools 


Omaha 


NE 


18 


1 


0 


Northeast 












Paterson City Schools 


Paterson 


NI 


0 


20 


0 


lersey City Public Schools 


Jersey City 


NJ 


0 


17 


0 


Bridgeport School District 


Bridgeport 


CT 


16 


0 


0 


Rochester City Schools 


Rochester 


NY 


20 


0 


0 


Springfield Public Schools 


Springfield 


MA 


15 


0 


0 


Providence School District 


Providence 


RI 


21 


0 


0 


Brockton 


Brockton 


MA 


12 


0 


0 


South 












Marion County Schools 


Ocala 


EL 


7 


8 


12 


Richardson Independent School District 


Richardson 


TX 


15 


0 


0 


Robeson County Schools 


Lumberton 


NC 


0 


0 


14 


McAllen Independent School District 


McAllen 


TX 


13 


0 


0 
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TABLE II.3 (continued) 



District Name 


City 


State 


Number Urban 
Schoolwide Title 
I Schools with at 
Least 60 Fifth 
Graders 


Number 
Suburban 
Schoolwide Title 
I Schools with at 
Least 60 Fifth 
Graders 


Number Rural 
Schoolwide Title 
I Schools with at 
Least 60 Fifth 
Graders 


Birmingham City 


Birmingham 


AL 


22 


0 


0 


Ector County Independent School District 


Odessa 


TX 


20 


0 


0 


Fulton County 


Atlanta 


GA 


1 


14 


3 


West 












Sacramento City Unified 


Sacramento 


CA 


18 


5 


0 


Tacoma School District 


Tacoma 


WA 


14 


0 


0 


Granite School District 


Salt Lake City 


UT 


0 


13 


0 


Oxnard Elementary 


Oxnard 


CA 


13 


0 


0 


Alhambra City Elementary 


Alhambra 


CA 


4 


8 


0 


Adams-Arapahoe (Aurora) 


Aurora 


CO 


14 


0 


0 


Albuquerque Public Schools 


Albuquerque 


NM 


24 


8 


0 



These changes improve the geographic diversity of the districts in the southern and western 
regions and the likelihood of having suburban and rural schools included in the study. lES agreed 
with the above recommendations, and recruiting has started with the districts recommended in Table 
II.2. 

Another school characteristic on which we will attempt to achieve some balance is English- 
language proficiency. Geographic distribution and a low-income student population may 
produce a reasonably diverse study sample. It is important to ensure that students with limited 
English proficiency (LEP) are also represented, however, because the Title I IRP singled out 
LEP students as a group that should be represented in the study. This implies that, to support 
subgroup analysis, a sizable fraction of schools should have some representation of LEP 
students. If possible, it would be desirable to have some schools with a small percentage of LEP 
students and some with a high percentage of LEP students so that we could observe whether 
interventions are effective for LEP students in each type of setting. We will consider this as we 

o 

talk with districts and schools about participating in the study. 



^ One factor that MPR considered as a possible criterion was whether the school was implementing Reading 
First, the federal government’s program to promote core reading skills in the early elementary grades. We believe 
that, if possible, it would be useful to observe the effectiveness of comprehension interventions in both types of 
settings — those with and without Reading First programs. For those schools that have Reading First programs, the 
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If we need to select back-up districts (beyond the initial 28 districts identified above) to 
reach our target number of schools, we will do so from the eligible pool of 157 districts noted 
above. We will attempt to select back-up districts to help ensure distribution both geographically 
and in key district characteristics. 

2. Recruiting Districts and Schools 

To make it easier to negotiate with districts and schools, we will begin recruitment efforts at 
the state level. State education officials can provide important contextual information on the 
school districts we plan to contact about participating in the evaluation. In addition, their explicit 
support may be able to sway district- or school-level officials and increase the likelihood that the 
districts and schools will participate. We will telephone the chief state school officers in the 
relevant states and discuss with them (or their designees) (1) the purpose of the study, (2) the fact 
that the burden of participation will be minimal, (3) the data to be collected and how they will be 
used, (4) our plans for protecting the confidentiality of school and student data, and (5) the 
potential benefits of participation. 

Some of the benefits we will stress, first to state officials and later to district and school 
officials, are (1) the opportunity for districts and schools to try new, promising approaches for 
improving the reading comprehension of low-income children; (2) teachers who are trained in 
these approaches with no out-of-pocket expenses for the school districts; and (3) the chance to 



(continued) 

proposed comprehension interventions may be seen as a follow-on, to continue with innovative practices that bring 
reading skills developed in early elementary grades to enhance those skills in the middle elementary grades. 
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contribute to a knowledge base with rigorous evidenee about what works in teaehing reading 
eomprehension — information that policymakers and educators across the eountry ean use.^ 

After we talk with state offieials and gain support from some or all of them, we will eontaet 
sehool districts. In the initial telephone ealls with sehool districts, we will introduee the study 
and diseuss its requirements. One important issue to diseuss with distriet offieials is parental 
eonsent. The Family Educational Rights and Privacy Act (FERPA) exempts studies eonducted 
under eontraet to ED (whieh aets as a representative of the Seeretary of Edueation) from needing 
parental consent for the eollection of sehool reeords. The study was ruled exempt from 
Institutional Review Board (IRB) review through the provision of Title 45 of the Code of Federal 
Regulations (CFR), Part 46 Proteetion of Human Subjeets, Section 101(B)(1), which exempts 
“researeh eondueted in established or eommonly aeeepted edueational settings, involving normal 
edueational praetices, sueh as (i) researeh on regular and speeial education instructional 
strategies, or (ii) researeh on the effeetiveness of or the eomparison among instruetional 
teehniques, eurrieula, or elassroom management methods.” However, our experienee indieates 
that some districts may nevertheless require parental eonsent. Plans for gaining parental eonsent 
will be one of several issues to be made explieit in the work plan that MPR will prepare for eaeh 
sehool district and school. The work plan, whieh will be agreed upon by loeal sehool offieials, 
will outline the roles of the researeh team and the sehool distriet and sehools. 

After gaining distriet-level approval to move forward and determining whieh schools to 
target, we will set up an in-person meeting with the distriet and sehool staff to provide distriet 
and sehool administrators with much the same information about the study as presented to state 

^ For example, information gleaned from the evaluation of reading comprehension interventions will fit the 
requirements for the What Works Clearinghouse funded by IBS. Our reports will present information that meets the 
requirements of data reporting as outlined by the Clearinghouse. 



27 




officials. A reading expert from the study team will also attend these meetings to provide 
information to districts and schools about the curricula. To participate in the evaluation, school 
officials must be receptive to random assignment and understand how it will be implemented. 
We base this statement on the approach MPR used in the Closing the Reading Gap study, a 
similar evaluation in which 50 schools across 27 school districts were randomly assigned to four 
intervention groups. That experience suggested that a critical aspect of the negotiation process 
with school staff was to clearly explain the random assignment procedures and the interventions. 

Our plan for implementing the experimental design, described in the next chapter, is based 
on that experience. First, we plan to maintain the integrity of random assignment by explaining 
to the school and district administrators why schools assigned to the control group should do 
nothing more than they would normally have done in the absence of the evaluation. They should 
neither adjust their current reading comprehension curriculum nor implement a new curriculum 
unless all schools in the district are doing so. This means that participating schools and fifth- 
grade teachers should continue implementing their current reading curricula, even if that means 
that some teachers are currently implementing another supplemental reading curriculum or using 
techniques that are part of the interventions the study is testing. The contrast the study is making 
involves the comparison of each intervention to the control group, which may include teachers 
who are teaching strategies similar to those being taught in the interventions the study is testing. 
Because random assignment of a school to a reading comprehension program is a necessary 
criterion for participating in the study, schools that refuse to participate in random assignment 
will be dropped from consideration. 

School officials must sometimes be persuaded of the benefits of participating in a study like 
this one. One potentially relevant benefit has to do with school accountability. Although not 
listed as an explicit criterion in the legislation authorizing the proposed Title I research, a 
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school’s failure to meet its Adequate Yearly Progress (AYP) targets could be an incentive for the 
school to adopt changes to its educational practices. 

Using information collected during the initial visits to districts and schools, we will develop 
a semifinal list of the districts and schools that are able and willing to participate in the 
evaluation. We expect the list will include about 13 to 15 districts. Out of these 13 to 15 
districts, we expect to reach final agreements to participate in the study with about 10 districts. If 
more than 10 districts agree to participate, we will assess the extent to which the districts 
contribute to the geographic diversity and face validity of the study, and make recommendations 
to IBS on the 10 districts we recommend including in the study. 
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III. EXPERIMENTAL DESIGN AND ANALYSIS 



The Evaluation of Reading Comprehension Interventions will use an experimental design in 
which schools are randomly assigned to selected reading comprehension interventions or to a 
control group. As part of the earlier reading comprehension intervention study design task that 
lES awarded to MPR, we made several recommendations concerning the design of the 
experiment to assess the impacts of up to four reading interventions. In this chapter, we expand 
on those recommendations by describing our general approach to meeting the major 
requirements of the evaluation, including indicating the research questions the study will address 
and how we will implement the experimental design and collect and analyze the data. 

A. RESEARCH QUESTIONS 

Table III.l lays out the main questions this study will address. The upper portion of the 
table shows the questions about the impact of reading comprehension interventions on students’ 
reading comprehension (both in general and when using expository text, such as that encountered 
in science and social studies). The lower portion of the table shows questions about conditions 
and practices (implementation and context) and their association with variation in impacts. The 
many questions related to the correlational analyses (associations) should not divert attention 
from the proposed evaluation’s most important goal — to document the impact of up to four 
interventions on students’ reading comprehension. 

We have also added a research question regarding the effect of the average intervention to 
the questions requested by lES in the original Statement of Work (SOW). We added this 
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ANALYSIS QUESTIONS, DATA ITEMS AND SOURCES, AND ANALYSIS METHODS 
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question because educators may want to know whether supplemental reading comprehension 
interventions, on average, can improve students’ reading comprehension/ 

B. THE EXPERIMENTAL DESIGN 

The experimental design of this study reflects (1) careful consideration of the questions 
posed by lES, the Title I IRP, and the TWG that worked with MPR on the design task; (2) our 
knowledge that all the interventions would be implemented in classrooms; (3) IBS’s suggestion 
that up to four interventions should be considered; (4) our recommendation that the study focus 
on the fifth grade; and (5) our concerns about contamination. To address these issues, we will 

implement a cluster randomized experiment consisting of about 100 schools divided evenly 

2 

among four intervention groups and a control group. 

We plan to include multiple school districts in the evaluation to reduce burden and achieve 
geographic diversity. Though this study is not designed to be nationally representative, 
including a geographically diverse sample of school districts provides some assurance that 
impacts measured by this study are not driven by idiosyncrasies in a few districts. We propose a 
design with 10 school districts, each contributing about 10 schools to the study. Next, we 
discuss the unit of random assignment, interpretation of the counterfactual, implementation of 
random assignment, and sample size and statistical power. 



* An alternative approach would be to conduct a global test of the joint significance of the four interventions. 
This test would be similar to testing the effect of the average intervention, because it would tell us if at least one of 
the interventions is significant without being able to identify which intervention is significant. 

^ Glazerman and Myers (2004) had proposed that the control group include 36 schools and that each 
intervention group include 16 schools. However, that proposal was based on the assumption of a large number of 
schools in a small number of districts. Because we have decided to target a larger number of districts (with fewer 
schools per district) to increase the external validity of the study, it is not possible to divide schools into groups 
using this ratio of treatment to control schools. For example, with 10 schools per district, we would need to assign 
3.6 schools per district to the control group and 1.6 schools per district to each treatment group, which is clearly not 
possible. 
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1. Unit of Random Assignment 

We will randomly assign schools, instead of classrooms or students, to treatment and control 
groups, for three reasons. First, randomly assigning interventions within schools would require 
that class formation occur before assignment of teachers to interventions. This could be 
impractical because training will occur before class rosters are finalized. Second, implementing 
more than one intervention in the same school would impose the unusual constraint on teachers 
that they cannot collaborate. This would create an unrealistic context for the implementation of 
the interventions, and it could also dissuade schools from participating in the study. Third, 
random assignment of classrooms within schools could lead to contamination. If classrooms or 
students in a single school were assigned to treatment and control groups, students in the control 
group could benefit from one of the treatments. For example, control group students might 
accidentally be included in treatment group activities, might talk with treatment group students 
about what they are learning in class, or could be reassigned to treatment group classrooms in an 
attempt to provide them with the intervention. Control group teachers might talk with treatment 
group teachers about the methods they are using to teach reading comprehension. By conducting 
random assignment at the school level, we greatly reduce these risks. 

2. The Counterfactual 

When interpreting the effect of the reading interventions, it is useful to consider the 
question, “Compared to what?” That is, what would students in each treatment group have 
experienced if they did not have access to their assigned reading intervention? Due to random 
assignment, the treatment groups and control group are, on average, statistically similar before 

^ In consultation with lES, we will explore the feasibility of conducting a small ethnographic study of the 
extent to which contamination might occur if random assignment were conducted within schools. Subsection 5 
describes this small study. 
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the intervention starts. Thus, we can infer what students in each treatment group would have 
experienced, on average, in the absence of their assigned intervention by examining the 
experiences of students in the control group. To accurately understand the counterfactual, we 
will use classroom observations to assess what students in the control group experience. Section 
C provides details on the classroom observations we plan to conduct. 

3. Implementing Random Assignment of Schools 

In principle, the random assignment of schools to interventions and a control group is 
straightforward. Figure III.l illustrates how the experimental design, which includes random 
assignment of schools within districts to interventions, will be set up. We will use a randomized 
block design, which is analogous to stratification techniques used to make statistical sampling 
more efficient. The primary blocking factor will be the school district. That is, we will conduct 
random assignment of schools within districts to hold district policies such as teacher hiring, 
compensation, and professional development constant. If feasible, we will also form two blocks 
of five schools within each district based on school characteristics (schools will be matched on 
reading test scores from the previous year).^’^ We will then randomly assign schools within 
those blocks to increase balance among intervention groups. 



When more schools are eligible to participate than can be included in the study from a given district, we will 
work with district administrators (who are most knowledgeable about the schools) to identify schools to invite to 
participate in the study. 

^ We will also examine other school characteristics such as the race and ethnicity of students in the schools, the 
percentage of students in the schools that are LEP, and whether schools are making AYP to determine whether those 
may be important blocking factors. For example, if all schools are roughly similar in terms of race and ethnicity, 
test scores, and percentage of LEP students, but one group of schools is making AYP and another group is not, we 
could form two blocks of five schools (making AYP and not making AYP) and conduct random assignment within 
those two blocks. 
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Random assignment will ensure that, when the data are pooled across districts, the only 
expected difference between the groups, on average, is that they will have been exposed to 
different interventions or to no intervention. Figure III. 1 also shows that we expect to include all 
fifth-grade students in each of the 100 schools in the study (we are assuming three fifth-grade 
classes with 26 students each in each school). 



FIGURE III.l 



RANDOM ASSIGNMENT OE SCHOOLS WITHIN DISTRICTS 



Intervention Group 




A 


B 


C 


D 


Control 


Within-district total of: 


Schools 


2 


2 


2 


2 


2 


Classrooms 


6 


6 


6 


6 


6 


Students 


156 


156 


156 


156 


156 


Study wide total of: 


Districts 


10 


10 


10 


10 


10 


Schools 


20 


20 


20 


20 


20 


Classrooms 


60 


60 


60 


60 


60 


Students 


1,560 


1,560 


1,560 


1,560 


1,560 



Protecting the Integrity of the Random Assignment Design. To protect the integrity of 
the random assignment design, evaluators must always guard against threats to internal validity. 
A key threat to the internal validity of the reading comprehension evaluation is “compensatory 
equalization.” This may occur if, for example, districts give additional resources for reading 
instruction to schools in the control group because they were not selected to participate in one of 
the four interventions, and those schools use these resources to modify reading, science, or social 
studies curricula or instructional practices. These modifications to the curricula in the control 
schools could boost students’ reading comprehension to levels above what would normally be 
expected in the absence of the interventions operating in other schools in the district. As a result, 
the evaluation could underestimate the impact of the reading interventions. 
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To help protect against the possibility that schools assigned to the control group will make 
compensatory adjustments that would threaten the integrity of the random assignment design, we 
will (1) carefully document the prevailing reading comprehension instruction and curricula for 
fifth graders before we announce the schools’ assignment status, and (2) monitor curriculum 
instruction to ensure that it does not substantially deviate from the prevailing curriculum. 
Specifically, as part of the implementation analysis, we will cross-check the data from our 
observations of experimental and control classrooms with our knowledge of what the curriculum 
would have been without the interventions. Limited changes in curricula that would have 
occurred in the absence of the study are permissible. If we observe more substantial changes in 
curricula, we will contact the school district to explain the importance of maintaining the 
integrity of the control group. We will also work with district and school administrators (both 
before the evaluation begins and again when the interventions start) to ensure they understand 
the consequences of making these types of adjustments. 

4. Sample Size and Statistical Power 

Due to the study’s size and use of rigorous methods, educators and policymakers may draw 
heavily on the findings of this study when choosing whether to adopt one of the interventions 
under consideration. Because of the policy relevance of the study, it is particularly important to 
make clear distinctions between effects that are likely to be real and those that are likely due to 
chance. 

When estimating sample size requirements for the design, we factored in four 
considerations: (1) the need to account for multiple comparisons when conducting tests of 
statistical significance, (2) whether we can estimate district fixed effects in the impact analysis, 
(3) the benefits of a baseline test to increase precision, and (4) IBS’s desire to detect differences 
between each intervention group and the control group equal to an effect size of 0.25 or larger 
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with a high probability. Furthermore, we worked from key assumptions concerning the expected 
number of fifth-grade classes within schools, the number of students within classes, and the 
expected correlation between pre- and post-test measures of reading achievement. These 
statistical considerations and assumptions suggest an experimental design that includes 20 
schools in each of the intervention groups (4 interventions x 20 schools = 80 total intervention 
schools) and 20 schools in the control group. 

a. Multiple Comparisons 

In this study, making clear distinctions between effects that are real and those that are due to 
chance is complicated by the issue of multiple comparisons. By comparing multiple intervention 
groups to a control group, and multiple treatments with each other, the probability that one of 
those differences will appear to be statistically significant is greater than the probability that any 
single difference will appear statistically significant. Intuitively, this is similar to the difference 
between the probability of a single toss of a coin yielding heads and the probability that at least 
one of several coin tosses will yield heads. 

The literature suggests a variety of techniques for addressing the issue of multiple 
comparisons. However, no clear consensus exists on how to address the issue of multiple 
comparisons. To calculate the MDE of the study, we will adjust for multiple comparisons using 
methods that strongly control the family-wise error rate (FWE). The EWE is the probability of 
falsely declaring at least one of the effects under consideration to be statistically significant. 
Specifically, we will use the Dunnett critical values for MDE calculations involving comparisons 
of the intervention groups to a control group, and the Tukey Honest Significant Difference 
(HSD) critical values for calculations involving all pair-wise comparisons among the 
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intervention groups.^ Appendix B describes these and other procedures for adjusting for multiple 
comparisons. 

b. District Fixed Effects 

Districts included in this study will not be chosen to be representative of any larger 
population of school districts. If the study were repeated, the same districts would be included, 
meaning that it may be appropriate to treat district effects as fixed rather than random.^ By 
treating district effects as fixed, the variance of the impact estimate may be substantially lower 
than if district effects were treated as random, which will allow us to detect smaller impacts. 

To calculate district fixed effects, we will need at least two schools per treatment condition 
in each district. If possible, we will recruit 10 schools from each district, yielding 2 schools in 
each of the 4 intervention groups and 2 schools in the control group. If we can attain this sample 
target, we will be able to treat district effects as fixed in our impact analysis. 

However, if we cannot recruit 10 or more schools in every district, we will have insufficient 
sample to calculate district fixed effects. If this occurs, we will attempt to approximate district 
fixed effects by pooling schools across similar districts. Under this scenario, we could form 
blocks consisting of matched pairs of districts with fewer than 10 schools. Districts would be 
matched based on geographical considerations and average baseline test scores. Each district 
with 10 or more schools would form its own block. In the impact analysis, we would calculate 
fixed effects for each block and random effects for the schools in each block. In this case, the 

® Adjusting for multiple comparisons using the Dunnett procedure increases the MDE by approximately 0.03 
standard deviations relative to no adjustment for multiple comparisons, and adjusting using the Tukey procedure 
increases the MDE by approximately 0.04 standard deviations relative to no adjustment for multiple comparisons. 

^ When treated as fixed, the effects of districts are estimated using district-indicator variables. The explanatory 
power of these variables reduces the amount of unexplained variation, thereby increasing the precision of impact 
estimates. 



39 




block fixed effects would not explain as much of the total variation as district fixed effects would 



have, because schools from two different districts will be included in some of the blocks. 

c. Using a Baseline Test to Increase Precision 

A large proportion of the variance in the test score that will be administered at the end of the 
study can be explained by students’ prior achievement. By estimating impacts with a regression 
model that uses prior achievement as one of the covariates, we can substantially increase the 
precision of the impact estimates, allowing us to detect smaller effects. However, the first test 
administered to students will not be a true baseline test - it will be administered shortly after the 
intervention has begun. Consequently, it might not be appropriate to include this early test in the 
analysis (see section D), which will adversely affect the study’s MDEs. 

An alternative approach to increasing the precision of the impact estimates is to use prior 
achievement at the school level as a covariate in the analytic model. Bloom et al. (1999 and 
2005) demonstrate that using prior achievement data aggregated to the school level can be almost 
as effective as individual student data in reducing the estimated MDEs for a study. ^ Specifically, 
they found only a 0.01 difference in the MDE between using student- level test score data and 
school-level data. The MDE calculations presented below assume that student level data will be 
used. If we determine that it is only appropriate to use school-level data on achievement (such as 
the test score data gathered from school records) due to the timing of the baseline test, we expect 
the MDEs will be slightly higher than those reported here. 



* Bloom et al. (2005) used data from Atlanta, Georgia; Columbus, Ohio; Houston, Texas; Newark, New Jersey; 
and Rochester, New York, on students in grades 3, 8, and 10. Bloom et al. (1999) used data from Rochester, New 
York, on students in grades 3 and 6. 
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d. Detecting the Desired Effect Size 



To assess appropriate sample sizes for the evaluation, we adopted the precision standard 
specified in the statement of work of an MDE of 0.25 standard deviations for the impact that is 
the primary focus of this study — the difference in student outcomes between each intervention 
group and the control group. Table III. 2 presents the MDE for three types of comparisons: 



1. Effect of Each Intervention. In the first column, we calculate the MDE when 
comparing each individual intervention group to the control group. We use the 
Dunnett procedure to adjust for comparing multiple intervention groups to a control 
group. This comparison answers the question that is the primary focus of this 
study — “Which interventions are effective?” 

2. Average Effect of All Interventions. In the second column, we calculate the MDE 
when comparing the average treatment group to the control group. This is a single 
comparison and therefore is not adjusted for multiple comparisons.^ This comparison 
answers the question, “What is the average effect of the supplemental reading 
interventions offered in this study on reading comprehension?” 

3. Differences in the Effects of Interventions. In the third column, we calculate the 
MDE when comparing differences between treatment groups. We use the Tukey 
HSD procedure to adjust for all pair-wise comparisons among intervention groups. 
This comparison answers the question, “Which intervention is most effective?” 



If we achieve the target sample size of 100 schools distributed evenly among 10 districts, we 
will be able to detect, with a high probability, an effect size of 0.25 or larger for each individual 
intervention. The MDE corresponding to the average effect of all interventions will be 0.17, and 
the MDE corresponding to all pair-wise comparisons among intervention groups will be 0.26 
(see panel A of Table III. 2). Analyses of district subgroups (for example, districts with a high or 
low percentage of EEP students) will only be able to detect effects larger than the target of 0.25, 



^ As noted earlier, although the original SOW did not specifically request this comparison, we propose that the 
comparison be made, because educators may be interested in whether supplemental reading comprehension 
instruction, in general, can improve students’ reading comprehension. 
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except in the case of the effect of the average intervention, where we are likely to detect an effect 
size of 0.24.^° 

If 100 schools can only be included by recruiting 5 schools from each of 20 districts, it will 
not be possible to reduce variance by controlling for district fixed effects. If this occurs, we will 
approximate district fixed effects by pooling schools across similar districts. Consequently, the 
MDE associated with comparing each individual intervention group to the control group is 0.31. 
The MDE corresponding to the average effect of all interventions will be 0.21, and the MDE 
corresponding to all pair-wise comparisons among intervention groups will be 0.32 (see panel B 
of Table III.2). 



TABLE III.2 

MDEs ON STUDENT TEST SCORES 



MDE, by Type of Comparison 



Effect of Each Average Effect of All Differences in the Effects 

Sample Size Intervention Interventions of Interventions 

A. 100 schools, 10 school districts, ^ divided evenly among 4 intervention groups and a control group 

Full sample 0.25 0.17 0.26 

50 percent subgroup of districts 0.36 0.24 0.38 

B. 100 schools, 20 school districts, divided evenly among 4 intervention groups and a control group 

Full sample 0.31 0.21 0.32 

50 percent subgroup of districts 0.45 0.30 0.47 



“With two schools per treatment condition per district, district fixed effects can be calculated. 

‘^With only one school per treatment condition per district, fixed effects are calculated for each of 10 blocks of two districts. 

Note: These calculations take into account clustering effects at the school level, as well as adjustments for multiple comparisons. A Dunnett 

adjustment is made for column 1, while a Tukey HSD adjustment is made for column 3. Because school districts are purposefully 
selected, we treat their effects as fixed. Because all fifth-grade classrooms at each school are included in the study, classroom 
clustering does not affect the variance of the estimated impact. The equation used to calculate the MDE between two curriculum 
groups is: 



Factor [a, /3,df)* * 



, 2{l-p) 
S N 



where: 

S is the number of schools in each group, N is the number of students, cif =0.05, = 0.20 (that is, 80 percent power), p (=0.10 in panel 

A, = 0.16 in panel B) is the between-school variance as a percentage of the total variance of the outcomes based on previous studies, 
and is the degrees of freedom, which is equal to the number of schools minus the number of strata minus the number of intervention 
and control groups. Previous impact evaluations have found that an of about 50 percent may be an appropriate assumption when 
baseline measures of test scores are available. Parameter assumptions are based on authors’ calculations using data from the LESCP 
and the analysis of past studies in Schochet (2005). 



*** Because the sample of students is large, analyses of 50 percent student subgroups will be able to detect 
effects almost as small as the main impacts. 
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5. An Ethnographic Study of Classroom Random Assignment 

The primary motivation for randomly assigning schools, rather than classrooms within 
schools, to intervention groups is the possibility of contamination of the control group. 
However, the extent to which contamination might actually occur has not been tested. If 
contamination does not occur, the statistical precision of random assignment studies could be 
increased considerably by assigning classrooms, rather than schools, to intervention and control 
groups. In the present study, randomly assigning classrooms rather than schools would reduce 
the MDE, when comparing multiple intervention groups to a common control group, from 0.25 
to 0.20. Alternatively, we could maintain an MDE of 0.25 but reduce the size of the study from 
100 schools to 70 schools. 

Because of the potential benefits of classroom random assignment, we propose to conduct, 
in consultation with lES, a limited study of contamination when classrooms within schools are 
randomly assigned to intervention and control groups. The study would involve a small number 
of schools in addition to the 100 included in the main study. We recommend identifying the 
schools by identifying an additional school in 8 of the 10 participating districts in the study. This 
process will result in eight schools in the ethnographic study, two schools in each treatment 
group. Eifth-grade classrooms in these schools would be randomly assigned to the intervention 
and control groups. 

To examine contamination, we could use teacher interviews, student focus groups, and 
limited student testing. We would ask teachers and students about specific details of the 
intervention being offered in their school, to determine whether control group students have been 
exposed to that intervention. It is important to talk with both treatment and control groups to 

** It will not be necessary to field all four interventions in a single school since the purpose of this smaller 
study is to study contamination, not to calculate impacts. 
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allow us to better understand whether contamination occurred. For example, the interpretation of 
a finding of no control group students having knowledge of the intervention strategies would be 
affected if the study were to also find that no treatment group students report knowledge of the 
strategies. In addition to determining whether the students have been exposed to the programs, 
we will try to understand how the exposure occurred. For example, separate focus groups of 
control group and treatment group students would: 



• Ask about the extent to which students talk with students in other classes about what 
they are learning in class. 

• Ask about the extent to which students work with students from other classes on 
projects or homework. 

• Assess whether students have seen the materials associated with the intervention 
being implemented in their school. If students have seen the materials, focus group 
facilitators will ask students how they came in contact with the materials (for 
example, through their teacher or other students). 

• Assess whether the students have heard of the techniques being taught as part of the 
intervention. If students have heard of the specific techniques being taught, focus 
group facilitators will ask how students heard of the techniques (for example, through 
their teacher or other students). 



Interviews of treatment group and control group teachers will assess: 



• The extent to which they generally collaborate with their colleagues on the teaching 
of reading. 

• The extent to which they have talked with colleagues about the specific intervention 
being tested in that school, including which aspects of the intervention were 
discussed. 

• The extent to which they have looked at or used materials from the intervention, 
including which materials were used. 

• Whether they believe restrictions on collaboration for the sake of a study would be 
acceptable to them. 



Students in both intervention and control classrooms could also be administered a brief test 
in spring 2007 (for example, a worksheet asking questions about the specific strategies being 
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taught as part of the intervention). If students in the control group do as well on the test as 
students in the treatment group, contamination would be suspected. At the request of lES, we 
can produce a more complete design for this smaller study, including survey instruments, 
interview protocols, and cost estimates. 

C. DATA COLLECTION 

To address the central research questions, we will obtain several types of information from a 
variety of sources during the 2006-2007 school year. Data collection activities will include (1) 
administering reading tests to students; (2) extracting school records data; (3) observing 
classrooms; and (4) obtaining information from principals, teachers, and curricula developers. 
These sources will provide information on reading comprehension instruction and curriculum, 
the context in which the instruction occurs, the implementation of the interventions, teacher 
characteristics, student characteristics, and student outcomes. Table III. 3 presents a general 
overview of the study’s data collection plan. It shows some of the major variables or concepts to 
be addressed under each domain shown in the conceptual framework in Chapter I. The rest of 
this section describes each of the data collection activities. 

1. Reading Tests 

The evaluation’s key outcomes are reading comprehension test scores in general reading and 
in expository text, such as the text students encounter in social studies and science. We will 
administer tests at the beginning and end of the 2006-2007 school year. Fall test administration 
will begin about three weeks after the start of the year (allowing approximately one week for 
enrollment to stabilize and one week for the dissemination of passive consent forms and parents’ 

After considering the preliminary impacts detected during the initial year of data collection and the 
percentage of students continuing in the same school or district in the 2007-2008 academic year, we may collect 
data for a second year. 
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TABLE III.3 



OVERVIEW OE DATA COLLECTION PLAN 





Reading 

Tests 


Student 

Records 


Classroom 

Observations 


Review of 
Materials 


School 

Information 

Form 


Teacher 

Survey 


Developer 

Survey 


1 Reading Comprehension Instruction and Curriculum | 


Instructional components 






X 


X 






X 


Types of material used 






X 


X 






X 


Fidelity 






X 


X 






X 


Quality 






X 


X 






X 


Frequency and duration of 
use 






X 


X 






X 


1 Context 1 


Teacher demographics, 

background 

characteristics 












X 




Teacher experience and 
training before study 
period 












X 




Professional development 
for selected interventions 






X 






X 




Presence of other staff 
(paraprofessional, literacy 
coach, reading specialist) 
to assist teacher or 
students 






X 






X 




School-level student 
characteristics 










X 






Professional culture at the 
school 












X 




Relevant state policies 










X 






Types of materials used 
for reading and social 
studies 








X 








1 Implementation | 


Fidelity 






X 










Quality and focus of 
student reading materials 






X 


X 








Cost 








X 






X 


Teachers’ knowledge and 
use of reading 
comprehension 
instructional practices 






X 






X 




1 Student Characteristics | 


Demographics 




X 












English-language ability 
(e.g., LEP status) 




X 












Use of learning support 
services 




X 












Eligibility for free or 
reduced-price lunch 




X 












Reading fluency 


X 














1 Outcomes | 


Reading comprehension 
(in general and when 
using expository text) 


X 














Grades 




X 












Attendance 




X 
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1 T 

reply). The fact that this fall test is administered after the intervention has begun has 
implications for the design and analysis (discussed in Section D). The spring test administration 
will begin about six weeks before the end of the school year (dates may vary slightly, depending 
on districts’ schedules for administering their own tests). 

For general reading comprehension, we will administer the Passage Comprehension subtest 
of the Group Reading Assessment and Diagnostic Evaluation (GRADE). The GRADE is a 
multiple-choice, paper-and-pencil, group-administered test that can be used to measure baseline 
skills and student improvement in critical reading areas. The Passage Comprehension subtest 
measures the student’s ability to comprehend an extended text as a whole. We will administer 
the GRADE to students at the beginning and end of the 2006-2007 school year. 

To assess students’ reading comprehension of expository text in science and social studies, 
we will use a test that either the Northwest Evaluation Association or the Educational Testing 
Service will develop from its item bank and from other newly created items. Half of the students 
will take the test in science, and half will take it in social studies. Eor this assessment, we are 
considering a computerized adaptive test (a test that adapts the questions that individual test- 
takers will receive based on whether they answered previous questions correctly), which requires 
fewer questions to determine a test-taker’s score. Students will take this test near the end of the 
2006-2007 school year, generally within the same week (but not on the same day) in which they 
take the GRADE test. 

We will also administer a brief fluency test — PRO-ED’ s Test of Silent Contextual Reading 
Eluency (TOSCRE). This group- administered test is used to assess students’ silent reading 
fluency — it measures skills such as word identification, word meaning, and sentence structure, 
all of which are important skills for reading comprehension. It will provide more refined data for 
If a district requires active consent, fall test administration will be delayed somewhat. 
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the subgroup analysis than would sorting students on the baseline reading comprehension test 
alone. We will administer this test along with the baseline administration of the GRADE test. 

2. Student Records 

The student records form will contribute useful background information on student 
characteristics and reading support services received, as well as on student outcome variables 
such as course grades (a draft is provided in Appendix C). We will use several methods to 
collect school records data. Some schools or districts may be able to extract the information 
electronically from their databases. At other schools, administrative staff will complete the 
records forms manually. In some cases, where confidentiality policies do not preclude access to 
student records by nonschool personnel, we will train local field staff to collect the student 
records information. 

3. Classroom Observations 

Our main approach for measuring implementation of the reading comprehension 
interventions in the treatment group schools and students’ reading instruction in control group 
schools is to observe classroom instruction. To provide context for interpreting the observations, 
we will also collect data on the nature of the professional development activities for each 
intervention and will ask reading experts to review each intervention’s curriculum for adherence 
to research-supported principles. Specifically, we will examine: 

• Variations in fidelity of implementation in treatment classrooms (across 
interventions and across schools and/or districts) 

• Variations in quality and content of reading comprehension instruction (across 
the four interventions and the control conditions) 

We will conduct two rounds of classroom observations — the first from November through 
mid-December, the second from mid-January through February. We selected the fall dates to 
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allow teachers to become familiar with the program before being observed. We selected the 
spring dates to avoid conducting observations during periods when schools typically administer 
standardized tests. During each round, we will observe all classrooms once and a randomly 
selected half of the classrooms a second time the next day. This approach — using clustered 
observations in the fall and spring, with repeated measurements at each point for half of the 
classrooms — will allow us to evaluate the reliability of estimates from classroom observations 
and assess temporal stability. These factors are particularly critical for the Likert scales being 
used for this study, as they require observers to make judgments about the quality of teacher 
behaviors. Because observer judgment is involved in these scales, they require higher levels of 
inference than scales that involve simply quantifying occurrences of relevant teacher behaviors, 
such as measures developed by Foorman and Scatschneider (2003) (see, for example, Gersten et 
al. 2005). 

We will observe reading and either science or social studies instruction in the classroom. 
Observations will last about 120 minutes. Staff will schedule observations of intervention 
classrooms on a day when teachers are conducting a science or history lesson using a textbook 
(as opposed to conducting assessments, science experiments, American history projects, etc.). 
This will let us observe treatment teachers using their assigned intervention and control teachers 
providing their regular text instruction. In both intervention and control classrooms, observers 
will complete a quality-of-instruction form that collects data on the use of research-based reading 
comprehension and vocabulary development techniques (see below for more information on this 
form). A new form module will be completed for each 15-minute interval of the observation 
(forms will be completed during the time treatment teachers are using the interventions, as well 
as during other aspects of their lessons). In treatment classrooms only, observers will also 
complete a second form that collects information on the fidelity of the implementation of the 
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intervention — that is, the extent to which teachers implement the curriculum approach as 
intended by developers (see below for more information on the fidelity forms). 

The instruments to be used were adapted from other instruments that were based on current 
reading research (for example, Edmonds and Briggs, 2003). Extensive testing is being done to 
ensure that the measures are reliable and sensitive to the quality and content of reading 
comprehension. When we train staff to complete the observation forms, we will assess inter- 
rater reliability to ensure that forms will be completed consistently across observers (preliminary 
testing found 0.90 inter-rater reliability). Eor staff who are unable to provide ratings consistent 
with other observers’ ratings, we will either provide additional training or train new staff. Next, 
we discuss the two forms staff will complete during classroom observations. 

Fidelity of Implementation. We will collect data on the fidelity of implementation in each 
intervention classroom, because a description of the fidelity of implementation is essential for 
understanding the relationship between an intervention and its impacts. Eidelity measures do not 
purport to assess the quality of instruction, but they do suggest why impacts may vary across 
districts and indicate if an intervention’s components are too difficult or confusing to be 
implemented widely. 

In developing these fidelity measures (current drafts are provided in Appendix C), we 
solicited extensive input from the developers to ensure that the measures reflect the critical 
elements of each intervention. To the extent possible, we are designing these measures to have a 
parallel structure across approaches — that is, they are similar in the number of items, the range of 
the Eikert-type scale, and how observations of teachers and students are recorded. The measures 

Fidelity of implementation (also known as treatment fidelity or treatment integrity) refers to the extent to 
which an intervention is implemented as intended across the entire duration of the evaluation (Gersten et al. 2000; 
Gersten et al., in press; Gresham et al. 2000). 
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address such questions as: Was the implementation carried out for the required amount of time? 
Were the required materials used? Were all the key features of the intervention (for example, 
teacher modeling, teacher/student think-alouds, peer groupings, peer discussions) implemented? 
For each statement, the observer will indicate either the presence or absence of an element or the 
extent to which the teacher implemented that aspect of the intervention (or the extent to which 
students were actively engaged in a task) on a Likert-type scale (for example, little, some, most; 
few — less than 25 percent, many — 25 percent to 75 percent, most — more than 75 percent). 

Quality Measure. We will collect data on the quality of instruction in all classrooms 
(treatment and control). The quality measure documents the quality of instruction based on 
current research on effective practice in reading comprehension. The measure (a current draft is 
provided in Appendix C) collects data on instruction procedures and strategies that are associated 
with, or are hypothesized to be linked to, enhanced outcomes in reading comprehension (Baker 
et ah, in press; National Reading Panel 2000; Snow 2002; Gersten et al. 2001). Items address (1) 
instructional quality, (2) frequency of instructional practices associated with improved reading 
comprehension outcomes, (3) student-generated questions, and (4) natural use of feedback and/or 
quality of prompting provided to support student learning. Most items are measured by 
frequency (counting occurrences), and several items are measured by Likert scales (such as 
measures of teachers’ responsiveness to students, which are rated as poor, fair, good, or 
excellent). 

4. School Information Form, Teacher Survey, and Developer Survey 

School Information Form. We will ask principals to complete a brief form on school-level 
characteristics. This form will provide background information on each participating school (see 
Appendix C). Items include school-level test scores for fifth graders for the most recent year, 
fifth-grade reading curricula (core and supplemental), and student characteristics (such as the 
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percentage of students with LEP and the percentage eligible for free or reduced-price lunch). We 
will mail this short, one-page preliminary information form to principals in spring 2006; data 
from this form will be used to plan the random assignment process and in some analyses. We 
will ask schools to complete a more comprehensive, two-page school information form as part of 
the student records collection process in spring 2007. In both cases, we will conduct telephone 
followup for nonresponse. 

Teacher Survey. The teacher survey will provide classroom- and school-level context for 
interpreting the reading assessment results (see Appendix C for the draft survey). The survey 
includes items from teacher professional culture scales (The Consortium on Chicago School 
Research 1999; Carlisle 2003), which can rapidly portray conditions affecting quality of 
instruction; an abbreviated version of a teacher efficacy scale (Hoy and Woolfolk 1993; Gibson 
and Dembo 1984), which correlates with the ability to benefit from professional development 
(Sparks 1988); and items on the teacher’s educational and professional background. If fifth- 
grade students have more than one teacher during the school day, we will ask all teachers 
providing instruction using the intervention — such as teachers of English/language arts, science, 
or social studies — to complete a survey. We will mail the teacher surveys in October and 
conduct telephone followup for nonresponse. 

Developer Survey. We will collect information from developers on the costs of their 
programs for schools using an ingredients approach (Eevin 1983). In this approach, we will 
identify all the items schools would need to purchase to implement and obtain support for the 
interventions. We will ask developers to specify the unit costs for each of the items, and we will 
calculate total costs per reading comprehension program based on the quantities needed of each 
unit. Analyses of these data will provide uniform information on the costs of large-scale 
implementation of each reading intervention. 
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D. ANALYSIS 



Next, we describe our general approach for conducting analyses of the quantitative data and 
synthesizing the quantitative and qualitative data. We discuss five topics: (1) estimating impacts, 
(2) estimating impacts for subgroups, (3) addressing anticipated complications, (4) examining 
long-term effects and duration of interventions, and (5) synthesizing quantitative and qualitative 
data. 

1. Estimating the Impacts of Reading Comprehension Interventions 

Because the evaluation will use an experimental design, the estimator of the effect of 
reading comprehension interventions is the difference between the average test scores for 
students in schools using the selected interventions and the average test scores for students in the 
control schools. For each of the test scores of interest (for example, scores on the GRADE 
comprehension test, and scores on the tests of comprehension of expository text in social studies 
and science), we will compute four separate impact estimates: (1) intervention A versus control, 
(2) intervention B versus control, (3) intervention C versus control, and (4) intervention D versus 
control. We will also compare the average test scores for students in all the intervention groups 
to those for students in the control group. 



In general, it is assumed that the interventions affect only the average achievement of students within the 
group. Some interventions may also affect the spread of achievement scores. For example, an intervention may 
have a larger impact on students at the lower end of the achievement distribution and thereby change the variance of 
the group and influence the mean. Such a finding can have both substantive and statistical implications. 
Substantively, it gives us more information about the effectiveness of an intervention. Statistically, it creates issues 
about how best to conduct tests of statistical significance. If we find differences in variances across interventions 
and control groups, we will need to use more complex analytic procedures to estimate the correct standard errors of 
the impact estimates. With large numbers of schools, such as proposed for the evaluation, it is appropriate to use the 
Huber- White or sandwich estimator for standard errors (Raudenbush and Bryk 2002). 

As noted earlier, although the original SOW did not specifically request this comparison, we propose that the 
comparison be made, because whether supplemental programs in general can affect reading comprehension may be 
of interest to educators. 
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In practice, the estimation of impacts will use hierarchical linear modeling (HLM), which 
allows us to take into account the clustering of students that occurs with random assignment of 
schools to interventions and the blocking that will be used when randomly assigning schools 
within districts to interventions (Raudenbush and Bryk 2002). Failure to take into account the 
clustering of students within schools can lead to overstating the statistical significance of 
impacts. We will also use techniques that take into account stratification and clustering using 
Taylor series linearization for simple comparisons of means, or generalized estimating equations 
(GEE) for regression models, because these approaches can be more stable when sample sizes 
are small. 

In addition to accounting for the clustering and the randomized blocks design used to define 

1 8 

the experiment, we will include baseline test scores as a covariate in the statistical models. 
Baseline test scores are strongly correlated with later test scores, and their inclusion as a 
covariate in the model will increase the precision with which we can estimate impacts. In fact, 
the use of baseline scores is critical in the design of the evaluation because it allows us to use a 
smaller number of schools than would be the case in a design where pretest scores were not 
available. 

The basic HEM is a two-level model. The first level corresponds to the student; the second, 
to the school. As expressed below: 



The GEE method is a generalization of generalized linear models (GEM) that takes into account within- 
group correlations between observations. 

** Because the fall 2005 test will be administered after the intervention has begun, it may be preferable to use 
baseline achievement at the school level, which will be collected using the school records form. Bloom et al. (1999 
and 2005) show that school-level data is nearly as effective as student-level data in reducing the MDE of a study 
when random assignment occurs at the school level. In Section 3, we discuss the trade-offs involved in using 
baseline achievement data from the school records form versus data from the fall test administration. 
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Level One: Students (i) within schools (j): 



Level Two: Schools (j): 

10 

= Yoo^j + Yoi^j + YoiCj + Yo3^j + YoaCONTj + '^A^^DISTj^^ + 

10 
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The Level One model relates students’ post-intervention test scores to their fall test scores, a 
constant term, and a residual term (unexplained variation)/^ The Level Two model relates the 
school-level average test score to the four treatment indicators (A, B, C, and D), the 
indicator for the control group for schools (CONT), fixed block effects (DIST), and a residual 
term that accounts for random variation in the school means. The Level Two model also 
allows the effect of baseline test scores (A;) ^ by district and by a deviation (^j^. ) 

from the overall effect ( Toj ). 

The parameters of interest to the evaluation are the ones that pertain to intervention impacts. 
The impacts are defined as the difference between the average achievement score for the 
intervention group and the control group. For example, the impact of reading comprehension 
intervention A on reading achievement is defined as the average achievement for the reading test 
for intervention A minus the average achievement for the reading test for the students in the 



For the proposed analysis in the two-level HLM framework, we would “mean-center” preintervention 
achievement around the grand mean. 

We have used a no-intercept model specification for Level Two so that the parameters on intervention and 
control group indicators can be directly interpreted as the average outcome in each group. 
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control group. Using the HUM, we see that the impact is estimated from the following 
expression: 

~ Too ~ To4 ■ 

Similar expressions can be constructed to estimate the impacts of the other interventions. 
Differences in the effects of interventions can also be computed. For example, the difference in 
effects between intervention A and B is: 

^A-B ~ Too ~ To\ ■ 

The HUM estimates the model parameters and their associated standard errors by using 
maximum likelihood and related methods. Statistical significance will be calculated using those 
standard errors and by taking into account the issue of multiple comparisons. The primary 
multiple comparison adjustment will be a stepwise technique that controls the FWE while 
providing more statistical power than the Dunnett or Tukey methods used to calculate the study’s 
MDE. See Appendix B for a complete discussion of the issue of multiple comparisons. In 
tables, we will denote impacts that are different from zero at a 5 percent level of statistical 
significance using a single asterisk. 

2. Student Subgroup Analyses 

Subgroup analyses will be important for revealing who benefits most from specific reading 
interventions. When developing the data collection instruments, we identified groups of children 
that may be of particular interest to policymakers or educators to inform future decisions about 
the implementation of the reading interventions. Some of these groups may center on students’ 
reading abilities (for example, students who have weaker initial reading comprehension skills 
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and lower levels of fluency, data that the study will be collecting at baseline). Other groups may 
center on students’ background characteristics (for example, LEP students, students from low- 
income families). The subgroup analyses may show, for example, whether students with weak 
reading foundations — those with limited fluency and weak reading comprehension skills at 
baseline — benefit as much as students who enter the interventions with stronger foundations. 

One limitation of this specific subgroup analysis is that the administration of the fall test 
used to determine students’ existing abilities will occur after the intervention has already begun. 
If the intervention has large, immediate impacts on students’ rank in the distribution of reading 
comprehension ability (large enough to change the ability group in which the student is 
classified), this analysis might not be valid. However, the impact on students’ rank would have 
to be considerable to invalidate subgroups based on the early test score, particularly if the 
subgroups are large (for example, defined by test score tertiles). 

3. Complications to Be Anticipated in Impact Analyses 

The analysis of the impact data is likely to give rise to three complications. First, students 
might transfer from one intervention school to another, or between an intervention school and a 
control school. Second, students may drop out of the evaluation and generate missing data.^^ 
Third, baseline test scores are likely to be collected shortly after the interventions have begun, 
rather than before. 

Student Mobility. Some students attending schools in the control group may move to one 
of the intervention schools, or students in an intervention school could transfer to a control 
school. The statistical literature refers to such a problem as noncompliance with the 

Students may also change classrooms within a school. Since all classrooms in the same grade will be in the 
study, this type of mobility will not complicate most analyses. An exception is the analysis of the relationship 
between implementation fidelity and outcomes, because it will not be clear in which classroom to include students. 
We will address this issue by investigating the sensitivity of findings to whether students are included in their 
original classroom or the classroom to which they transfer. 
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experiment’s protocols (see, for example, Angrist et al. 1996). The effects of students’ actual 
intervention status will be estimated using the instrumental variables (IV) method, where initial 
assignment to intervention groups will be used as an instrument for the group that students 
actually participate in. 

The IV method is a two-step procedure. In the first step, students’ final intervention status is 
regressed on their initial intervention assignment, and the coefficient estimates from this 
regression are used to calculate students’ predicted intervention status. In the second step, the 
outcome is regressed on the predicted intervention status calculated in the first step. The 
coefficient on predicted intervention status in the second regression provides the estimate of the 
program’s impact on the outcome. This approach is analogous to dividing an impact by the 
participation rate of students assigned to a treatment group to generate the impact on participants. 

Missing Data. Although we anticipate high response rates to both surveys and student tests, 
it is reasonable to expect that some students will be “lost” from the evaluation, particularly if 
they transfer to a school outside of the study. Fortunately, we can adjust for “nonresponse” by 
using statistical procedures such as the estimation of nonresponse models. These models predict 
the probability of remaining in the study and responding to the survey or of being tested on the 
basis of student characteristics such as earlier test scores, sex, and race and ethnicity. 
Nonresponse models are used to create statistical weights that allow the remaining students to 
“stand in” for students who left the sample. These weights are applied while conducting the 
analyses and will produce unbiased estimates of intervention effects under the assumption that 
the observed characteristics of students adequately account for the processes that led to students 
leaving the sample. 

Baseline Test Scores. As noted above in Section B on the experimental design, the 
administration of the GRADE test in fall 2006 will take place after the interventions have begun. 
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Consequently, this will not be a true baseline test, and might already reflect an impact of the 
interventions on students’ scores on the GRADE test. If this early test does reflect an impact of 
the interventions, including it as a regressor in the impact analysis could understate the impact of 
the interventions. 

Consistent with the options laid out for handling this issue in the study design, two solutions 
to addressing this complication in the analysis are possible. First, we can include school-level 
average test scores from previous years (Bloom et al. 1999 and 2005). One drawback to this 
approach is that the tests administered by the state may not measure the same skills as the 
GRADE test that we will administer at followup, thereby reducing its ability to explain variation 
in the GRADE test administered at followup. Another drawback is that we would not be able 
to form subgroups based on students’ prior reading ability, nor can we use the baseline test to 
assess change in scores on the GRADE. 

A second solution to this problem is to include the early test scores in our analysis, despite 
the problem with the test’s timing. Although the early test will be administered after the 
intervention has begun, the administration will still take place early in the intervention period, 
and the test might not yet exhibit effects of the intervention. If we find that including the 
baseline test increases the precision of our impact estimates without attenuating the size of the 
impact, we will deem it appropriate to include the early test in the analysis. Alternatively, if 
including the early test reduces the size and significance of our impact estimates, we will 
conclude that it is not appropriate to include in the analysis. We will consult with lES to 
determine how we will address this issue. 

As described in Section C, we will also administer assessments of students’ reading comprehension of 
expository text in science and social studies in the spring, but not in the fall. Whether the GRADE would be a better 
predictor of the science and social studies tests than the state tests is unknown. However, since the GRADE test is 
designed to assess reading comprehension in general, students’ scores on the fall GRADE test may be a better 
predictor of their performance on the spring science and social studies tests than the state tests. 
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4. Long-Term Effects of Reading Comprehension Interventions and Duration of 

Interventions 

In the SOW for this project, lES stated two possible options for future data collection and 
analysis. Option 1 would address whether reading comprehension interventions in the fifth grade 
produce impacts that last beyond the intervention year by conducting additional student testing in 
spring 2008. As described in the original SOW, the study may examine second-year effects if 
three conditions are satisfied. The interventions must have been well implemented, they must 
yield substantially meaningful impacts after one year, and student mobility must be reasonably 
low. The rationales behind these criteria are straightforward. First, assuming that an 
intervention showed an impact after the first year but was poorly implemented, its replication 
would be difficult in other districts because it failed to follow a clearly specified approach. 
Second, if impacts are insignificant after one year, it is hard to imagine that meaningful impacts 
would occur in later years without an additional year of intervention. Third, without an adequate 
sample size or with strong differential attrition between the treatment and control groups, there 
may be too little statistical power to detect meaningful impacts, or the analysis may produce 
biased impact estimates. 

Option 2 would address issues related to the duration of the reading comprehension 
interventions. In this option, the study would follow some fifth graders into sixth grade and 
assess the impact of adding a second year of reading comprehension instruction. This option 
calls for us to randomly assign fifth graders (rising sixth graders) in the intervention schools to 
an intervention or control group within those schools. However, this approach runs the risk of 
contamination, where students in the control classrooms are indirectly influenced by the 
intervention, either through interaction with peers or through collaboration among teachers. 
Direct contamination effects are also possible if some teachers provide the intervention to 
students in the control group who the teachers view as being in particular need of help. The 
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pressure on teachers to provide students with whatever help they can may be particularly great in 
schools that are not meeting their AYP requirements. 

We will assess the feasibility of these options in consultation with IBS over the course of the 
study. At the request of IBS, we can provide a more detailed design for these options. 

5. Synthesizing Quantitative and Qualitative Data 

To help with the interpretation of impact estimates, we will describe, using data from the 
classroom observations, the basic characteristics of the four interventions and the reading 
instruction provided to control group students. We will also analyze how these characteristics 
correlate with impacts. Birst, we will use summary tables to describe (1) the school conditions in 
each intervention group and the control group, (2) the practices of teachers in each intervention 
group and the control group, and (3) the implementation fidelity in each intervention group. 
Second, we will relate these conditions and practices to intervention effectiveness using three 
different analytic approaches. We do not plan to relate implementation fidelity to intervention 
effectiveness, because there is no corresponding measure of implementation fidelity for the 
control group. An important limitation of all these analyses is that the conditions, practices, 
and fidelity measures that we observe might be correlated with many important factors that we 
do not observe. Therefore, these analyses are all potentially biased and will be interpreted 
cautiously. 



It would be possible to calculate the correlation between student test scores and implementation fidelity 
using only students in the intervention group. However, because implementation fidelity is a choice of teachers, it 
would not be possible to distinguish the effects of fidelity from the effects of unobserved teacher characteristics. Put 
another way, we cannot provide separate answers to the questions, “What is the effect of the intervention when it is 
implemented well?” and “What is the effect of a highly motivated teacher?,” because we observe both effects 
simultaneously. 
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a. Descriptive Analyses 

Descriptive analyses will draw on the data sources described in Section C, and the specific 
data items will be collected using the forms in Appendix C. Examples of the conditions, 
practices, and implementation fidelity measures are: 

• Conditions. Average test scores for fifth graders at each school in the previous year, 
fifth-grade reading curricula, percentage of students eligible for free or reduced-price 
lunch, race and ethnicity of students in the schools, school size, professional culture 
in the school, and the percentage of students classified as LEP. 

• Practices. Whether the teacher managed student behavior effectively, gave accurate 
and clear explanations, asked students to justify or elaborate their responses to 
questions, and modeled particular reading comprehension strategies before, during, or 
after reading. 

• Implementation Fidelity. Whether implementation was carried out for the required 
amount of time, correct materials were used, and key features of the intervention (for 
example, teacher modeling, teacher/student think-alouds, peer groupings, peer 
discussions) were used. 

We will compare the average conditions and practices in each intervention group and the 
control group using descriptive tables. The columns of the tables will correspond to each 
intervention and control group, and the rows will correspond to specific conditions and practices. 
The cells in each table will report the average condition or practice for each group. Similar 
tables reporting implementation fidelity will be prepared for each of the four intervention groups. 
Combined with the main impact findings, these tables will inform the development of hypotheses 
regarding why some interventions appear more effective than others. 

b. Correlational Analyses 

We propose three approaches for relating conditions and practices to impacts. The first 
approach relates the district-level impacts computed as part of the main impact analysis to 
district-level conditions and practices. The second approach calculates impacts in each district 
for the classrooms or schools that belong to subgroups defined by conditions or practices. The 
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third approach compares average outcomes across intervention and control groups for schools or 
classrooms that belong to subgroups defined by conditions or practices, regardless of school 
district. These approaches, and the trade-offs of the approaches, are described below. 

The first approach draws on the design of the experiment by relating district-level impacts to 
district- level characteristics. Each district in the study can be regarded as a mini-experiment 
with its own impact estimate. We can calculate correlations between those impact estimates and 
district-level characteristics, including student, classroom, and school characteristics aggregated 
to the district level. For example, we can find the correlation between impacts and the 
proportion of students in a district who are poor. District-level impacts are the most precise 
because they can be calculated accounting for district fixed effects and because they involve all 
schools, classrooms, and students in a district. District-level conditions and practices are the 
least refined, however, because of the high level of aggregation. 

A limitation of this approach is that a multivariate analysis will not be feasible due to the 
small number of districts, meaning that any correlation between impacts and conditions could be 
due to the influence of other factors that are correlated with impacts and the condition of interest. 
Furthermore, variation in conditions and practices aggregated to the district level could be 

24 

limited and may mask important variation within districts. 

A second approach involves calculating impacts for the subgroups of schools in each district 
that meet the condition of interest. For example, within each district, we could calculate the 
impact of an intervention for schools with a high proportion of low-income students by 
calculating the difference in average student test scores for students in high-poverty schools in 
the intervention group and students in high-poverty schools in the control group. These impacts 
can then be averaged across districts for an overall impact of the intervention in schools with a 
Aggregation of data could lead to biased estimates of the true associations (Freedman 1999). 
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high concentration of students from low-income families. The same approach would be used to 
estimate impacts for schools with a lower concentration of low-income students. We would then 
compare these two impacts to assess whether they are significantly different. This approach is 
appealing because it ensures that all districts are represented in both the treatment and control 
groups. However, this approach requires sufficient variation in school characteristics in each 
intervention group, within each district. In the above example, this would require a high-poverty 
school in each intervention group and in the control group in each district. This data requirement 
may be too stringent for analysis of school-level conditions, because there are likely to be only 
two schools per intervention group per district. Analyses of classroom-level conditions and 
practices might be feasible, however, because there will be, on average, six classrooms per 
intervention group in each district. 

The third approach allows schools and classrooms in different intervention groups to be 
compared across districts. For example, there might not be a school with a high proportion of 
low-income students in every intervention group in every district. Using this method, the 
average outcome for high poverty schools in the intervention group will be compared to the 
average outcome for high poverty schools in the control group, regardless of which districts 
these schools are in. Put another way, the impact for the high poverty subgroup will not be 
computed within district. This approach would be repeated for low-poverty schools, and then the 
impacts of the two groups would be compared to determine whether they are significantly 
different. Though this approach allows a more refined analysis of conditions and practices, the 
fact that impacts are calculated across districts reduces the precision of the impacts, because 
district effects must be treated as random. Furthermore, if the analysis of conditions and 
practices is too refined (for example, if very small subgroups are analyzed), large spurious 
differences between intervention groups could threaten the face validity of this analysis. 
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We will use all three of these methods to examine the relationship between program effects, 
conditions, and practices. When interpreting findings in study reports, findings that are robust to 
the choice of method will be reported with more confidence. All these findings will be 
interpreted cautiously, however, because all three approaches are fundamentally 
nonexperimental and may be influenced by correlations between unobserved factors and the 
conditions and practices we can observe. 
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APPENDIX B 

THE ISSUE OF MULTIPLE COMPARISONS 




The purpose of testing the statistical significance of the difference in outcomes between a 
treatment and control group is to determine whether the observed difference is most likely due to 
an effect of the treatment, rather than random chance. When more than one pair of groups is 
compared in this way (for example, when comparing multiple treatment groups to a single 
control group), the chances of falsely declaring at least one of the differences statistically 
significant is greater than when a single comparison is made. This is analogous to the difference 
in the probability of winning the lottery with a single ticket and the probability of winning the 
lottery with multiple tickets. This issue is known in the literature as the issue of multiple 
comparisons. 

For this study, we will need to make multiple comparison adjustments when calculating 
MDEs and analyzing impacts (see Chapter III). These adjustments will be made when 
conducting three distinct types of comparisons (1) when comparing several treatment groups to a 
single control group, (2) when making all pair-wise comparisons among intervention groups 
(excluding the control group) and (3) when estimating impacts for subgroups of students. 

The literature suggests two general approaches to addressing the issue of multiple 
comparisons. The first approach is to use adjustments that control the probability that at least 
one of the comparisons under consideration will be falsely declared significant. This probability 
is known as the family wise error rate (FWE). Most of the procedures developed to adjust for 
multiple comparisons are designed to control the FWE. 

More recently, an alternative to controlling the EWE has been suggested. This second 
approach is to control the proportion of "discoveries" (that is, impacts that appear significant) 
that are not truly different from zero. This proportion is known as the false discovery rate 
(EDR). The difference between the EWE and the EDR is that the EWE criterion considers it 
unacceptable to have any effects declared falsely significant, whereas the EDR tries to keep the 
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number of falsely signifieant effeets low relative to the number of eorreetly signifieant effeets. 



Our primary analyses will be foeused on eontrolling the more eonservative FWE, however we 
will explore how our findings ehange when using a teehnique that eontrols the FDR. 

In this appendix, we first deseribe the types of eomparisons we intend to make. We then 
deseribe several methods of adjusting for multiple eomparisons, weighing the advantages and 
disadvantages of eaeh method. We eonelude by deseribing the multiple eomparison proeedures 
that will be used on this study’s design and analysis of impaets. 

A. COMPARISONS PLANNED FOR THIS STUDY 

We will make three qualitatively distinet sets of eomparisons in this study that will require 
adjustment for multiple eomparisons: 



1 . Comparing Multiple Interventions to a Single Control Group. Determining whether 
eaeh intervention has a statistieally signifieant impaet is the primary foeus of the 
study. Edueators and polieymakers need to be able to draw upon these findings with 
eonfidence that reported impaets are real, and not due to ehanee differences among 
groups. Therefore, we believe it is necessary to report the statistical significance of 
these effects after adjusting for multiple comparisons. 

2. Making All Pair-Wise Comparisons Among Treatment Groups. Though this issue 
is important to educators and policymakers, the study is designed primarily to detect 
differences between intervention groups and a control group, not to detect differences 
among intervention groups. Since differences among intervention groups are likely to 
be substantially smaller than main impacts, a larger study would be needed to reliably 
detect these smaller effects. When interpreting findings in study reports, differences 
among treatment groups that are robust to multiple comparison adjustments will be 
reported with more confidence. (Differences that are significant only when no 
adjustments for multiple comparisons are made will be regarded as suggestive.) 

3. Calculating Impacts Among Multiple Subgroups. Another circumstance in which 
adjustments for multiple comparisons are sometimes considered is when calculating 
effects for different subgroups. We will report subgroup findings with and without 
adjustments for multiple comparisons. For example, if we were to compare all four 
intervention groups to the control group for girls and all four intervention groups to 
the control group for boys, we would need to adjust for eight comparisons. When 
interpreting findings in study reports, subgroup findings that are robust to multiple 
comparison adjustments will be reported with more confidence. (Findings that are 
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significant only when no adjustments for multiple eomparisons are made will be 
regarded as suggestive.) 

In addition to these three sets of eomparisons/ we will also make one eomparison that does 
not require adjustment. In this eomparison, we will eompare the average intervention to a single 
control group. Education administrators and policymakers will probably want to know if 
supplemental reading instruetion, in general, is effeetive at improving reading eomprehension. 
To address this question, the study can examine whether the effeet of the average intervention in 
this study is statistieally signifieant. Sinee this is a single eomparison, we will not adjust its 
statistical significance to aceount for multiple eomparisons, yielding a more powerful test. 

B. MULTIPLE METHODS THAT ADJUST FOR MULTIPLE COMPARISONS 

We consider six different approaches to adjusting for multiple eomparisons. The first five 
approaehes are designed to eontrol the EWE while the sixth approaeh eontrols the EDR. While 
the first five approaches all control the EWE, they differ in the settings in whieh they are 
designed to be applied and/or in terms of their statistieal power. The Bonferroni adjustment ean 
be used in any setting but has the least statistieal power. The Dunnett and Tukey adjustments are 
intended for more speeifie settings than Bonferroni and aehieve greater statistieal power by 
taking into account correlations among tests. The resampling approaeh can be applied in nearly 



* It could be argued that, because we are comparing all treatments to a control group and making all pair-wise 
comparisons among treatment groups, we should adjust for all these comparisons in both contexts. That is, we are 
essentially proposing to make 10 comparisons (4 treatment groups versus a control and 6 pair-wise comparisons 
among the treatment groups), yet we are proposing to make separate adjustments for only the 4 comparisons and 
then for the next 6 comparisons. We believe this separation is appropriate because the primary focus of the study is 
on the difference between each treatment group and the control group. The pair-wise comparisons are of secondary 
interest and are intended as suggestive findings that can be used to inform future research. 

^ Useful discussions and examples of these methods are available at [ http://www.tufts.edu/~gdallal/mc.htm 1 
and [ http://davidmlane.com/hvperstat/intro ANOVA.html l. Hochberg and Tamhane (1987) provide a thorough 
review of multiple comparison procedures in use at that time. Westfall and Young (1993) and Westfall (1997) are 
examples of further refinements of these procedures. 
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as many settings as Bonferroni while also taking into account correlations between tests. 
Stepwise procedures can be applied in a variety of setting and increase power by taking into 
account logical relationships between tests. 

1. A Conservative Approach: Bonferroni 

The Bonferroni adjustment is a simple, conservative approach that controls the FWE but is 
not very powerful. The adjustment is simply to multiply the unadjusted p-value of a t-test by the 
total number of comparisons being made. The appeal of this approach is that it is easy to 
implement in any setting and with little computational cost. It can be used both when comparing 
multiple treatments to a single control group and when making all pair-wise comparisons among 
groups. It can also be used to adjust for additional comparisons that are not easily accounted for 
using other techniques — for example, adjusting for all pair-wise comparisons among treatment 
groups using multiple outcomes and multiple subgroups. 

The disadvantage of this approach is that it does not take into account correlations among 
tests, resulting in a conservative adjustment. For example, when comparing multiple treatment 
groups to a single control group, the tests are correlated because they each involve the same 
control group. For example, the probability of falsely rejecting Hoi: Ta - CONT = 0 (where Ta 
represents the mean outcome for the first treatment group (intervention A) and CONT represents 
the mean outcome for the control group) is not independent of the probability of falsely rejecting 
Ho 2 -' Tb - CONT = 0, because both tests are based on the same control group. By ignoring this 
correlation, we reduce the probability of detecting a real effect more than is needed in order to 
control the FWF. For example, when comparing four treatment groups to a single control group, 
the Bonferroni adjustment would be to multiply the p-value from the test by 4. If correlations 
among tests were taken into account (see section on Dunnett below), the adjustment would be to 
multiply the p-value from the test by approximately 3.4. 
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2. Comparing Multiple Treatments to a Control Group: Dunnett 

The use of the multivariate t-distribution in the context of comparing multiple treatment 
groups to a common control group was first suggested by Dunnett (1955). By using the 
multivariate t-distribution, the Dunnett procedure controls for correlations between tests. An 
important assumption underlying the Dunnett test is that the test statistics actually follow the 
multivariate-t distribution. If the underlying distribution of the outcome of interest is heavily 
skewed, then this assumption is not valid (Westfall and Young 1993). 

3. All Pair-Wise Comparisons: Tukey’s Honest Significant Difference (HSD) 

The same general approach used by Dunnett to compare multiple treatments to a control 
group can also be applied when making all pair-wise comparisons among multiple treatment 
groups. That is, we can calculate critical values for these test statistics, taking into account the 
correlations between the tests using a multivariate t-distribution. This approach is named after 
Tukey and is sometimes called Tukey’s HSD. This approach is discussed in Hochberg and 
Tamhane (1987). As is the case with Dunnett, this approach relies on the assumption that the 
test statistics follow a multivariate-t distribution. 

4. Resampling Techniques 

Westfall and Young (1993) suggest using the bootstrap (or a permutation approach) to 
calculate p- values when making multiple comparisons. The primary appeal of this approach is 
that, instead of assuming test statistics follow a multivariate t-distribution, the joint distribution is 
estimated empirically. The approach is conceptually simple, requires fewer distributional 
assumptions than other methods, and controls the FWE. 

The idea of resampling techniques in general is to simulate what would happen if an 
experiment were repeated a large number of times (say, 1,000). For example, to calculate the 



B.7 




variance of the sample mean using a resampling method, we would draw 1,000 samples (with 
replacement) from our actual data. For each sample, we would calculate the sample mean, 
yielding 1,000 estimates of the mean. The variance of the sample mean would be the variance of 
those 1,000 estimated means. The appeal of this approach is that we do not have to make any 
assumptions regarding the distribution of the sample mean. Instead, we estimate the distribution 
of the sample mean directly, based on our data. 

In the case of multiple comparisons, we can use resampling to create 1,000 sets of test 
statistics corresponding to the comparisons of interest. We can then calculate the multivariate 
distribution of the test statistics using these 1,000 sets. This empirically estimated distribution 
can then be used to adjust for multiple comparisons rather than relying on the multivariate-t 
distribution. 

This approach is particularly useful when there is good reason to believe that differences 
between groups do not follow a multivariate t-distribution — for example, when the outcomes of 
interest have a particularly skewed distribution. In general, test scores are approximately 
normally distributed. In this study, however, scores could be skewed due to the makeup of the 
sample (children in low-achieving schools). If we find that the distribution of test scores is 
skewed, we will investigate whether our interpretation of findings changes when using multiple 
comparison adjustments based on resampling. 

5. Step-Wise Procedures 

The Dunnett and Tukey procedures described above are single-step procedures in which 
adjustments are applied equally to all the tests being performed. For example, if we compare 
group A to B, B to C, and A to C, the same adjustment is used for each comparison. 
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Step-wise procedures do not apply the same adjustment to each test. Instead, they use 



information about whether one test, or set of tests, was rejected to alter the multiple-comparison 
adjustment made to subsequent tests. 

Not all step-wise tests control the FWE. An example of a step-wise test that does not always 
control the FWE is a two-step procedure called Fisher’s ESD, also sometimes called “protected 
ESD.” In the first step, an F-test is used to evaluate the joint significance of all tests under 
consideration. If that test is rejected, all the individual t-tests are performed without any 
adjustment. Though intuitively appealing, this procedure suffers from a loophole. If all the null 
hypotheses are true (that is, there are no significant differences among groups), then this 
procedure controls the FWE. However, consider what happens if just one of the null hypotheses 
is false while the rest are true: the F-test is rejected due to the one false null, and all the other 
hypotheses are tested with no adjustment whatsoever. In other words, the FWE for those 
remaining tests is not controlled. Other approaches that seem intuitively appealing, but upon 
closer examination do not control the FWE under all combinations of null and alternative 
hypotheses, are Neuman-Keuls and the Duncan Multiple Range test. See Hochberg and 
Tamhane (1987) for details as to why these methods fail to control the FWE. 

Step-wise procedures that always control the FWE have also been developed. Hochberg and 
Tamhane discuss the theory behind these tests and explore several specific examples. Westfall 
(1997) and Westfall et al. (1999) provide further refinements to these procedures, as well as 
algorithms for implementation. For example, a step-wise procedure might rank the differences 
between all groups from the largest difference to the smallest difference. These procedures begin 



^ See Hochberg and Tamhane (1987), p. 3. 
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by applying a Tukey or Dunnett adjustment to the largest difference between groups, and then 
make less conservative adjustments to the remaining differences. 

Though useful when calculating impacts, these step-wise procedures are less useful in the 
context of MDE calculations in this study. The MDE analysis for this study is focused on the 
largest difference between groups, because if we cannot detect the largest difference, we will be 
unable to detect any of the smaller differences. Therefore, MDE analyses are based on the 
Dunnett or Tukey adjustments. 

6. A Procedure that Controls the FDR 

Benjamin! and Hochberg (1995) first suggested the EDR as an alternative to the EWE. They 
also suggested a step-wise procedure that controls the EDR. We will investigate the sensitivity 
of our findings to this alternative approach to adjusting for multiple comparisons. When 
describing findings in the study report, we will regard impacts that are significant when 
controlling the EWE with more confidence. (Eindings that are not significant when controlling 
the EWE but are significant when controlling the EDR will be regarded as suggestive.) 

C. CONCLUSION 

To calculate MDEs, we will rely on the single-step procedures of Dunnett (comparing 
multiple treatment groups to a single control group) and Tukey (making all pair-wise 
comparisons among treatment groups). Eor impact analysis, we will rely on step-wise 
procedures that always control the EWE, though we will also explore the sensitivity of our 
findings to procedures that only control the EWE under the null hypothesis of no effects 
(Eisher’s ESD) and methods that control the EDR. 
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APPENDIX C 

DATA COLLECTION INSTRUMENTS 




TEACHER SURVEY (2006-2007) 

NATIONAL EVALUATION OF READING COMPREHENSION PROGRAMS 

U.S. DEPARTMENT OF EDUCATION 



ATTACH LABEL HERE 
Teacher ID Teacher Name 
School ID School Name 



IF ABOVE INFORMATION IS INCORRECT-; 

PLEASE MAKE CORRECTIONS ERECTLY ON LABEL. 

This survey is part of the Evaluation of Readit^ Comprehension Programs, a national 
evaluation being conducted for the U.S. Department of Education. The questions ask 
about the training you received on the reading comprehension program, professional 
culture at your school, your reflections, and your background. All information you 
provide will be kept confidential. While you are not required to respond, your 
cooperation is needed to make the results of this survey comprehensive, accurate, and 
timely. Thank you. 



Please return the completed form to: 

Mathematica Policy Research, Inc. 

600 Maryland Ave., SW, Suite 550 
Washington, DC 20024-2512 
ATTN: Valerie Williams 



If you have questions, please contact: 

Valerie Williams 
Phone; 888.535.0283 
FAX: 202.863.1763 

E-mail: vwilliams@mathematica-MPR.com 



According to the Paperwork Reduction Act of 1995, no persons are required to respond to a coiiection of 
information uniess it dispiays a vaiid 0MB controi number. The vaiid 0MB controi number for this 
information coiiection is ####-####. The time required to compiete this information coiiection is estimated 
to average 20 minutes per response, inciuding the time to review instructions, search existing data 
resources, gather the data needed, and compiete and review the information coiiected. If you have any 
comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, piease 
write to: U.S. Department of Education, Washington, D.C. 20202-4651. If you have comments or 
concerns regarding the status of your individuai submission of this form, write directiy to: U.S. 

Department of Education, Institute for Education Sciences, Washington, D.C. 20208-5651 . 

0MB NO.: ####-#### 

EXPIRATION DATE: ##/##/200# 



DRAFT 
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SECTION I. READING COMPREHENSION PROGRAM TRAINING 



This section asks about the training you recently received on the reading 
comprehension program you are using in your classroom as part of the Evaluation of 
Reading Comprehension Programs. 

If you did not receive training in a reading comprehension program for this 
study, piease check this box and go to Question 4 on the next page. ► O 



1. Thinking about the initiai training you received on the reading comprehension program you are using 
with your ciass, how wouid you rate the foiiowing: 



In EACH ROW, CHECK ONE BOX ONL Y 


Poor 


Fair 


Good 


Excellent 


a. Trainer’s (or trainers’) knowiedge of reading 

comprehension instruction for fifth graders 


iD 




3D 


4D 


b. Trainer’s (or trainers’) preparedness 




iD 




3D 


4D 


c. Trainer’s (or trainers’) presentation styie 


a I 


I 




3D 


4D 


d. Quaiity of content covered in training 


0 1 


iD 




3D 


4D 


e. Amount of content covered in training 


11 / 


iD 




3D 


4D 


f. Training scheduie (i.e., amount 
various sessions) 


of time spent on the 


iD 




3D 


4D 


g. Materiais provided in training.... 






iD 




3D 


4D 



2. Overaii, how weii did the initiai training you received 
prepare you to use the reading comprehension 
program with your students? 



Not AT All Somewhat Very Well 
iO 2\Z\ sO 



3. if you have any other comments about the training, piease note them beiow. 
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SECTION II. PROFESSIONAL CULTURE 

This section asks about the professional culture within your school.^ 

4. Conversations About T eaching 



During the past school year, how often have you had conversations with colleagues about... 



In EACH ROW, CHECK ONE BOX ONLY 


Less than 
Once a Month 


2 OR 3 Times 
A Month 


Once or 
Twice A Week 


Daily 


a. 


The goals of this school? 


iD 




3D 


4D 


b. 


Development of new curriculum? 


iD 




3D 


4D 


c. 


Managing classroom behavior? 


iD 




3D 


4D 


d. 


What helps students learn best? 


iD 




3D 


4D 



5. My Grade Level 



How much do you disagree or agree with each of the following? 








In EACH ROW, CHECK ONE BOX ONLY 




Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


a. Teachers in this grade level trust each 


other 


iD 




□ 

CO 


4D 


b. It's OK in this grade level to discuss feelings, 
worries, and frustrations with other teachers 


iD 




□ 

00 


4D 


c. Teachers respect other teachers who take the 
lead in grade level improvement efforts 


iD 




□ 

00 


4D 


d. Teachers in this grade level respect those 

colleagues who are expert at their craft 


iD 




□ 

00 


4D 


Please notice different response choices for the 


Not At 






A Great 


ITEM BELOW. 






All 


A Little 


Some 


Extent 


e. To what extent do you feel respected by other 
teachers in this grade level? 


iD 




□ 

00 


4D 



Please notice different response choices for the 

ITEM BELOW. 


None 


About 
Some Half 


Nearly 
Most All 


f. How many teachers in this grade level really care 
about each other? 


□ 

o 


2^ 


□ 

□ 

00 



Items in this section are from The Consortium on Chicago School Research. (1999). “Improving Chicago's Schools: 
The Teachers' Turn, 1999; Elementary School Teacher Survey, 1999.’’ Chicago, IL. Available at [www.consortium- 
chicago.org.] 
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6. Access to New Ideas 

How often have you... 



In 



a. 

b. 



c. 



d. 



e. 



f. 



EACH ROW, CHECK ONE BOX ONL Y 



3 TO 4 5 TO 9 1 0 OR More 
Never Once Twice Times Times Times 



Taken courses at a college or university 
relative to Improving your school? 

Participated In a network with other teachers 
outside your school? 

Discussed curriculum and Instruction 
matters with an outside professional group 
or organization? 

Attended professional development 
activities organized by your school (Include 
meetings that focus on Improving your 
teaching)? 

Attended workshops or courses sponsored 
by your school district (exclude requi 
services)? 

Attended professional development 
activities sponsored by the teachers' 
union? 




oQ 1 


□ 




3D 


4D 


5D 


oQ 1 


□ 




3D 


4D 


5D 


oQ 1 


□ 




3D 


4D 


5D 


oQ 1 


□ 




3D 


4D 


5D 


oQ 1 


□ 




3D 


4D 


5D 




□ 




3D 


4D 


5D 



7. My Experience of Change 

How much do you disagree or agree with the following? 



In EACH ROW, CHECK ONE BOX ONLY 


Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


a. 


Most changes Introduced at this school 
Involve only a few teachers; rarely does the 
whole faculty become Involved 


iD 




□ 

CO 


4D 


b. 


We receive adequate professional 
development support for the changes we 
Introduce at our school 


iD 




□ 

CO 


4D 


c. 


Most changes Introduced at this school gain 
little support among teachers 


iD 




□ 

CO 


4D 
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8. Professional Development 



How much do you disagree or agree with the foiiowing? 



Overaii, my professionai deveiopment 
over the past schooi year. . . 

In EACH ROW, CHECK ONE BOX ONLY 


experiences 


Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


a. 


...have inciuded opportunities to work productiveiy 
with teachers from other schoois 


1 


□ 




3D 


4D 


b. 


...have inciuded enough time to think carefuiiy 
about, to try, and to evaiuate new ideas 


1 


□ 




3D 


4D 


c. 


...have deepened my understanding of subject 
matter 


1 


□ 




3D 


4D 


d. 


...have heiped me understand my students better 


1 


□ 




3D 


4D 


e. 


...have been sustained and coherentiy focused, 
rather than being short term and unreiated 


1 


□ 




3D 


4D 


f. 


...have inciuded opportunities to work 
with coiieagues in my schooi 


productively 


1 


□ 




3D 


4D 


g- 


...have ied me to make changes in my 


teaching... 


1 


□ 




3D 


4D 


h. 


...have been cioseiy connected to my i 
improvement pian 


school's 


1 


□ 




3D 


4D 





Strongly 


Strongly 


Check one box onl y 


Disagree Disagree Agree 


Agree 


i. Most of what 1 learn in professional development 
addresses the needs of the students in my 
classroom 


□ 

CO 

□ 

CM 

□ 


4D 
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9. Leadership and Support 



How much do you disagree or agree with the foiiowing? 

Strongly 

In each row, check one box only Disagree Disagree Agree 

a. The principai at this schooi is strongiy committed 

to shared decision making iQ 2^ s i I 

b. The principai at this schooi works to create a 

sense of community in the schooi iQ 2^ s i | 

c. The principai at this schooi promotes parent and 

community invoivement in the schooi iQ sQ s | | 

d. The principai at this schooi supports and 

encourages teachers to take risks iQ sQ s | I 

e. The principai at this schooi is wiiiing to make 

changes iQ sQ s | | 

f. Most changes introduced at this schooi receive 

strong support from the principai iQ sQ s | | 

g. The principai at this schooi encourages teachers 

to try new methods of instruction i Q 2^ s i | 



Strongly 

Agree 

4D 

4D 

4D 

4D 

4D 

4 D 

4D 
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10 . Thoughts About Teaching Reading^ 



How much do you agree or disagree with the foiiowing? 



In EACH ROW, CHECK ONE BOX ONLY 


Disagree 


Disagree 


Agree 


Agree 


a. 


I feei I need to make changes in the methods I use to teach 
chiidren to read and speii 


iD 




3D 


4D 


b. 


I get heip from staff members to understand some chiidren’s 
difficuities iearning to read 


iD 




3D 


4D 


c. 


I have benefited from opportunities to iearn more about 
methods for teaching reading 


iD 




3D 


4D 


d. 


The chiidren in my ciass are making satisfactory progress in 
iearning to read 


iD 




3D 


4D 


e. 


I do not have sufficient materiais to teach reading effectiveiy 




1 


3D 


4D 


f. 


I do not understand why some chiidren iearn to read easiiy 
whiie other chiidren struggie to iearn basic reading skiiis 


£□1 




3D 


4D 



Strongly 



Strongly 



g. The iiteracy coach supports my efforts to teach reading 

effectiveiy 

If A LITERACY COACH IS NOT AVAILABLE 






THIS OUESTION AND CHECK THIS BOX ► □ 1 


iD 




3D 


4D 


h. 


I have a good understanding of how chiidren acquire ianguage 
and iiteracy skiiis 


iD 




3D 


4D 


i. 


I wish I had more opportunities to discuss how to teach reading 
with other teachers 


iD 




3D 


4D 


j- 


I feei I am good at teaching reading and writing 


iD 




3D 


4D 


k. 


The principai of my schooi supports my efforts to teach reading 
effectiveiy 


iD 




3D 


4D 


i. 


I wouid iike to iearn methods to heip chiidren deveiop their orai 
ianguage 


iD 




3D 


4D 


m. 


I iook for opportunities to iearn effective methods to teach 
reading and writing 


iD 




3D 


4D 


n. 


I couid do a better job teaching reading if I had more assistance 
from aides or voiunteers in my ciass 


iD 




3D 


4D 


0. 


I know how to assess the progress of my students in reading .... 


iD 




3D 


4D 


P- 


The parents of chiidren in my ciass support my efforts to teach 
their chiidren to read 


iD 




3D 


4D 


q- 


The schooi day is organized to maximize instructionai time 


iD 




3D 


4D 



2 

Items on this page were borrowed from Joanne Carlisle’s “Teacher's QUEST: Self-Administered Questionnaire” 
(Regents of the University of Michigan: Ann Arbor, Ml, 2003), with minor modifications. 
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SECTION III. TEACHER REFLECTIONS 

This section asks for your reflections.® 



11. Teacher Reflections 



In EACH ROW, CHECK ONE BOX ONLY 



a. How much can you do to control 
disruptive behavior in the classroom? . 

b. How much can you do to motivate 

students who show low interest in 
schoolwork? 

c. How much can you do to get students 

to believe they can do well in 
schoolwork? 

d. How much can you do to help your 

students value learning? 

e. How much can you do to get children 

to follow classroom rules? 

f. How much can you do to calm a II 
student who is disruptive or noisy*.... 

g. How much can you use a variety of 

assessment strategies? 

h. How much can you assist families in 

helping their children do well in 
school? 

Please notice different response 

CHOICES FOR THE ITEMS BELOW. 



i. To what extent can you craft good 

questions for your students? 

j. To what extent can you provide an 

alternative explanation or example 
when students are confused? 

Please notice different response 

CHOICES FOR THE ITEMS BELOW. 



k. How well can you establish a 

classroom management system with 
each group of students? 

l. How well can you implement 

alternative strategies in your 
classroom? 



Very Quite a A Great 

Nothing Little Some Bit Deal 



iD 


□ 

CO 

□ 

CM 


4D 


5D 


iD 


□ 

CO 

□ 

CM 


4D 


5D 


iD 


□ 

CO 

□ 

CM 


4D 


5D 


iD 

i 


2n sD 


4D 


5D 




□ 

CO 

"□ 


4D 


5D 


f 

P 


sO 


4D 


5D 


iD 


□ 

CO 

□ 

CM 


4D 


5D 


iD 


□ 

CO 

□ 

CM 


4D 


5D 


Not AT 


Small Moderate 


Quite A 


A Great 


All 


EXTENT Extent 


Bit 


EXTENT 



1D 


□ 

CM 


CO 

□ 


4D 


5D 


1D 


□ 

CM 


□ 

CO 


4D 


□ 

in 


Not AT 
All 


Slightly 


Moderately 


Quite 

Well 


Extremely 

Well 


iD 


□ 

CM 


□ 

CO 


4D 




iD 


□ 

CM 


□ 

CO 


4D 


5D 



3 

Items on this page were borrowed from W.K. Hoy and A.E. Woolfolk’s “Teachers' Sense of Efficacy Scale” 
(Elementary School Journal, vol. 93, pp. 355-372), with minor modifications. 
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SECTION IV. BACKGROUND 



This section asks about your background. 

12. How many years have you taught, either full-time or part-time, at the elementary or secondary 
level (not counting the current school year)? Include years teaching in both public and private 
schools. Do not include time spent as a student teacher. 

I I I T OTAL YEARS TEACH I NG 



13. How many years have you been teaching in THIS school (not counting the current school 
year)? If you have had a break in service of one year or more, please report the year that you 
returned to this school. Do not include time spent as a student teacher. Include years spent teaching 
both full- and part-time at this school. 



I Total years teaching at this school 



14. What grade levels have you taught? Che^ 
1 Qlst grade 



2 Q 2 nd grade 
sQSrd grade 
4Q4th grade 
sQsth grade 



eQeth grade 



yQzth grade 
sQsth grade 
gQsth grade 
loQlOth grade 




12th grade 
13I I Ungraded 
14I I Kindergarten 
is| I Prekindergarten 



15. For each degree below, please check Yes or No in Column A to indicate if you hold that 
degree. For those degrees you hold, please specify your major field of study in Column B 
and the year you received the degree in Column C . 



In each row, check one box in Column A. If 
YOU ANSWER Yes, complete Columns Band 

C FOR THAT ROW. 


A. Degree 
Held 


B. Major 


C. Year 
Received 


Yes No 


a. Associate’s degree 


□ 

0 

□ 




1 






1 


b. Bacheior’s degree 


1 1 1 ol 1 












c. Master’s degree 


□ 

0 

□ 












d. Educationai speciaiist or professionai 






dipioma (at ieast one year beyond a 






1 






1 


master’s degree) 


1| 1 ol 1 












e. Certificate of Advanced Graduate Studies... 


□ 

0 

□ 












f. Doctorate (Ph.D., Ed. D.) 


□ 

0 

□ 












g. Professionai (M.D., D.D.S., J.D., L.L.B) 


□ 

0 

□ 













Prepared by Mathematica Policy Research 



C.12 



Teacher Survey 2006-07 





16. Which of the following describes the teaching certificate you currently hold in this state? 

Check one only 

1 □ Regular or standard state certificate or advanced professional certificate 

2 \Z\ Probationary certificate (the initial certificate issued after satisfying all requirements except the 
completion of a probationary period) 

sD Provisional or other type given to persons who are still participating in an “alternative 
certification program” 

4Q Temporary certificate (requires some additional college coursework and/or student teaching 
before regular certification can be obtained) 

sD Emergency certificate or waiver (issued to teachers who do not have regular certification who 
need to complete a regular certification program in order to continue teaching) 



17. In what content area does the teaching certificate marked above allow you to teach in this 
state (e.g., elementary general, secondary general, special ed., a specific subject matter)? 

Content Area 



18. Column A: Please indicate if you participated in any professional development activities 
listed below in the past 12 months. 

Column B: If you mark “yes” in Column A, please indicate in Column B how many hours you 
spent on the activities. Include courses you have taken for recertification or advanced certification, 
workshops sponsored by your district, conferences, or other training that is relevant to your teaching. 



In each row, check one box in Column A. 
If you answer Yes, check one box in 
Column B. 


A. Participated? 


B. 


Number OF Hours 


Yes 


No 


8 OR 
Fewer 


9-16 


17-32 


33 OR 
More 


a. 


Reading instruction 


iD 


□ 

O 


iD 




□ 

CO 


4D 


b. 


Science instruction 


iD 


□ 

o 


iD 




□ 

00 


4D 


c. 


Social studies instruction 


iD 


□ 

o 


iD 




□ 

CO 


4D 
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1 9. Are you male or female? 

1 □ Male 
2 \Z\ Female 

20. Are you of Hispanic or Latino origin? 

1 nVes 
on No 



21 . How do you describe yourself? [O-ECXALLTmi apply) 
1 0 American Indian or Aiaska Native 

2\Z\ Asian 

sO Biack or African American 
4n Native Hawaiian or Other Pacific Isiander 
sG White 

22. What is your year of birth? 

I I I I I Year 




CONTACT INFORMATION 

Piease provide your contact information and the best time to reach you in case we have questions about 
your responses. 



Mr./Ms. First Name 



Last Name 



Street 


Apt. Number 




City 


State 


Zip 


E-mail ADDRESS 






L ] 






Phone Number (Include Area Code) 



Best TiME to reach you 



THANK YOU FOR COMPLETING THIS SURVEY 
FOR THE U.S. DEPARTMENT OF EDUCATION. 
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PRELIMINARY SCHOOL INFORMATION FORM: National Evaluation of Reading Comprehension Programs 



School 


Prindoal: 




Person comoleting form: 




Phone number 


I I 



1. How many students are enrolled 

a. In this school ? 

b. In the fifth grade ? 

2. How many fifth-grade classes de yeu have? | | | Fifth-grade classes 

3. What percentage of your school's students are: 

a. Eligible for the federally funded free or reduced-price lunch program? | | |% of students 



b. Classified as limited English proficient (LEP)? | | | | % of students 

4. What resources does your school use for its fifth-grade reading curriculum? 



Core Curriculum 


^Hfslame 


Publisher 


Textbook 






n a I 




Basal reader series 






K 




Special program 










Supplemental Curriculum 








Soecifv tooic (e.g., ohonics): 






Soecifv tooic (e.g., ohonics): 







Total enrollment 
Fifth-grade students 



5. Please complete the table below for the most current average reading and math standardized test scores for this 
school’s fourth- and fifth-grade students . 



Grade 

Level 


Test 


Publisher 


Month/ 

Year 


Reading 


Math 


Standard 

Score* 


Natl 

Percentile 


Standard 

Score* 


Nat’l 

Percentile 


4th 
















4th 
















5th 
















5th 
















*lf standard scores are not available, check here if reporting: 


1 □ Scaled Scores 
2 \~\ Raw Sceres 


1 □ Scaled Scores 
2 n Raw Scores 



6. How many students enrolled in this school are: 

a. Hispanic or Latino? | | | | | Students 

b. Not Hispanic or Latino? I I I I I Students 



7. How many students enrolled in this school are: 

a. American Indian or Alaska Native? 

b. Asian? 

c. Black or African American? 

d. Native Hawaiian or other Pacific Islander? .. 

e. White? 



Students 

Students 

Students 

Students 

Students 



Please return this form to Mathematica Policy Research, Inc. in the postage-paid envelope provided or by 
faxing it to 202-863-1763, attention Meiissa Dugger. Thank you very much. 
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SCHOOL INFORMATION FORM (2006-2007) 
National Evaluation of Reading Comprehension Programs 



INSERT SCHOOL LABEL HERE 



1 . For what grade levels does this school offer instruction? (Check All T hat Apply) 



1 □ Prekindergarten 
2^ Kindergarten 
sO 1st grade 
4 □ 2nd grade 



5 □ 3rd grade 9 □ 7th grade 

6 □ 4th grade 10 □ 8th grade 

7 □ 5th grade n □ Other (specify): 

8 □ 6th grade 12 O^ngraded (including ungraded special ed. students) 



2. What was the total number of students enrolled 
in this school around the first of October 2006? . 




L 



Students enrolled 



3. How many students were absent on the most 
recent school day? 



Students absent 



4. During the 2005-2006 academic year or summer, approximately 

how many students transferred from your school? | | | | Student transfers 

Exclude students who graduated to a middle or high school. 



5. What percentage of students in the 2006-2007 academic year are: 

a. Eligible for the federally funded free or reduced-price lunch program? | | | | % of students 

b. Classified as limited English proficient (LEP)? I I I I % of students 



6 . 



How many students enrolled in this school are: 

a. Hispanic or Latino? 

b. Not Hispanic or Latino? 



Students 

Students 



7. How many students enrolled in this school are: 



American Indian or Alaska Native? 










Students 


Asian? 










Students 


Black or African American? 










Students 


Native Hawaiian or other Pacific Islander? 










Students 


White? 


LL 






Students 
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8. How many teachers are employed at this school? | | | | Teachers employed 

Include all full- and part-time regular classroom teachers, special 
area or resource teachers, long-term substitute teachers, and 
itinerant teachers. Do not include student teachers, teachers’ 
aides, or short-term substitute teachers. 

9. How many fifth-grade students were enrolled in 

this school around the first of October 2006? | | | | | Fifth-grade students 

10. How many fifth-grade classes do you have? | | | Fifth-grade classes 

1 1 . What type of school is this? (Check one) 

1 □ Regular 

2 Q Special Program Emphasis (science/math school,^ talented/gifted school, foreign language 
immersion school, etc.) 

sO Special Education (primarily serves students with disabilities) 



4^ Other (specify): 

12. Does this school offer a magnet program? i GYes oONo 

13. Is this a charter school? i OYes oONo 

14. a. Is this a Title I school? i OYes o^No 

b. /f yes; Is it schoolwide Title I? i QYes o[^No 



15. Is your school participating in any comprehensive school reform? 
1 □ Yes^ Please describe: 



oO No 



16. Please complete the table below for the most current average reading and math standardized test 
scores for this school’s fourth- and fifth-grade students. 











Reading 


Math 


Grade 

Level 


Test 


Publisher 


Month/ 

Year 


Standard 

Score* 


Natl 

Percentile 


Standard 

Score* 


Natl 

Percentile 


4th 
















4th 
















5th 
















5th 
















*lf standard scores are not available, check here if reporting: 


1 □ Scaled Scores 

2 □ Raw Scores 


1 □ Scaled Scores 

2 □ Raw Scores 



Please return this form to Mathematica Policy Research, Inc. in the postage- 
paid envelope provided or by faxing it to 202-863-1763, attention Melissa Dugger. 

Thank you very much. 
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STUDENT RECORDS FORM (2006-07) 

EVALUATION OF READING COMPREHENSION INTERVENTIONS 



Student bar-coded label 



SECTION I. STUDENT ENROLLMENT, ATTENDANCE, AND CHARACTERISTICS 



1 . What is this student’s date of birth? 

2. Is this student male or female? 

3. What is the student’s ethnicity? 



J/l I I/I 



4. What is this student’s race? 
(Check All T hat Apply) 



Month Day Year 

Maie 2 ^ Femaie 

Hispanic or Latino 
oQ Not Hispanic or Latino 
Don’t know 

!□ American Indian/Aiaska Native 
2 Q Asian 



nP 



Q Biack or African American 
□ Native Hawaiian or other Pacific Isiander 



3 

4 

5 □ White 

Don’t know 



5. What was this student’s enrollment status on the iast day of the 2006-07 schooi year ? (check one) 
If the student transferred, was expelled, or left for another reason, please fill in the box to the right. 

1 □ Enroiied at this schooi on the iast day 

of the 2006-07 schooi year 

2 □ Transferred to another schooi ^ 



3 □ Expeiied 

4 □ Other (Specify)_ 






Last day of attendance: 


1 l/L 


I/I 1 


Month 


Day Year 


Name of new schooi: 






New schooi’s address: 




City 


State 



6. How many days was this student absent during the 2006-07 schooi year? (write “0” if no absences) 

a. I I I I Total days absent in the 2006-07 schooi year 

b. I L_l I Unexcused days absent in the 2006-07 schooi year (write “NA” if not available) 



7. How many times and days was this student given an in-school or out-of-school suspension from this 
schooi during the 2006-07 schooi year? (write “0” if this student received no suspensions. If your 

SCHOOL DOES NOT OFFER IN-SCHOOL SUSPENSIONS, WRITE “NA” IN 7 FOR TIMES AND DAYS SUSPENDED.) 
NUMBER OF TIMES SUSPENDED TOTAL NUMBER OF DAYS SUSPENDED 

a. I I I Number of times suspended in-schooi I I I I Days suspended in-schooi 

b. I I I Number of times suspended out-of-schooi I I I I Days suspended out-of-schooi 

8. Is this student. . . (check one in each row) 

YES NO 

a. Ciassified as limited English proficient (LEP)? i □ o □ 

b. Eiigibie for the federaiiy funded free or reduced-price lunch program? i □ o □ 



According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of Information unless It displays 
a valid 0MB control number. The valid 0MB control number for this Information collection Is ####-####. The time required to 
complete this Information collection Is estimated to average 30 minutes per response. Including the time to review Instructions, search 
existing data resources, gather the data needed, and complete and review the Information collected. If you have any comments 
concerning the accuracy of the time estlmate(s) or suggestions for Improving this form, please write to: U.S. Department of Education, 
Washington, D.C. 20202-4651 . If you have comments or concerns regarding the status of your Individual submission of this form, write 
directly to: U.S. Department of Education, Planning and Evaluation Services, Washington, D.C. 20208-5651 . 
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9. For which of the following disability categories has this student been officially identified? 



(CHECK ALL THAT APPLY) 

1 □ Autism 

2 □ Deaf-blindness 

3 □ Developmental delay 
4Q Emotional disturbance 



6 □ Learning disability 

7 □ Mental retardation 
sD Orthopedic impairment 
gD Other health impairment 



11 □ Traumatic brain injury 

12 □ Visual impairment 

13 □ Other disability (Specify): 



5 □ Hearing impairment io □ Speech or language impairment i4 □ None of the above 



SECTION II. SERVICES 

1 0. Which of the following services does this student receive in reading ? (check all that apply) 

1 □ Reading support 

2 □ Speech/Language support 

3 □ English as a Second Language (ESL)/English for Speakers of Other Languages (ESOL), English 

Language Development (ELD) 

4 □ Any other extra support or tutoritjj^jg., Title I or other extra help to bring students up to grade-level 

expectations) 

5 □ None of the above 

1 1 . Does the student have an Individual Education Plan (lEP) or a 504 Service Agreement? 

(CHECK ALL THAT APPLY) 

1 □ Student receives services under an lEP 

2 □ Student receives services under a Service Agreement 

3 □ Neither of the above ► GotoOuestion 16 

12. Does the lEP or Service Agreement provide specifications about the student’s reading Instruction? 

(CHECK ONE) 

1 □ Yes 0 □ No — ► Go TO Question 1 6 




13. What is the total amount of reading Instruction specified in the lEP or Service Agreement? 

I I I I MINUTES/DAY OR I I I % OF DAY OR □ No time is Specified 

14. In what areas does the lEP or Service Agreement specify reading goals or objectives? 

(CHECK ALL THAT APPLY) 

1 □ Vocabulary 4 □ Accurate and fluent reading of connected text 

2 □ Decoding, word attack strategies s □ Other (Specify): 

3 □ Comprehension e □ None specified 



15. Does the lEP or Service Agreement specify that the student should receive any of the following 

reading-related accommodations? (check all that apply) 

1 □ Material read aloud s □ Additional time on assignments and/or tests 9 □ None of the above 

2 □ Tests read aloud e □ Modified spelling (spelling errors not graded) 

3 □ Shorter passages i □ Other (Specify): 

4Q Out-of-grade-level material sD Other (Specify): 
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SECTION III. GRADE LEVEL AND COURSE GRADES 



1 6. In what grade was this student enrolled in the 2006-07 school year? Grade 



1 7. Has this student been promoted to the next grade for the 2007-08 school year? (check one) 
If the student will attend a new school next year, please fill in the box to the right. 

1 □ Yes — ► Promoted to grade: 
oD No 

9n Don’t know 



If attending a new school next year: 

Name of new school: 

New school’s address: 

City State 



1 8. Was this student ever retained in a grade? (check one) 
1 □ Yes 

oD No ► Go TO Question 20 

9D Don’t know — ► Go to Question 20 




1 9. If yes, in what grades was this student retained? (check all that apply) 

1 □ Prekindergarten 4 □ 2nd grade e □ 4th grade 

2 □ Kindergarten s □ 3rd grade i □ 5th grade 

sD 1st grade 



20. In which of the following classes was this student enrolled for all or part of the 2006-07 school year? 
For those classes in which the student was enrolled, indicate the level of the class and the student’s 
final grade in the class. (If a separate grade is not provided for class, enter “NG.”) 

Please include one copy of this school’s grading scheme — a blank report card with an 
explanation of the grading scheme (for example, showing S=satisfactory or A=95-100) or some 
other grade key explaining the meaning of grades used— when you return the completed 
student records for your school. 



Subject 


A. Enrolled 


B. Class Level 


C. Grades 




Yes 


No 


Advanced 


Regular 


Remedial 


Special Ed. 






1. Fall 














1 ST Qtr 


2nd Qtr 


a. Reading 


iD 


oD 


iD 




3D 


4D 


1 1 


1 


b. English/Language 


iD 


oD 


iD 




3D 


4D 


1 


1 1 


c. Math 


iD 


oD 


iD 




3D 


4D 


1 1 


1 


d. Science 


iD 


oD 


iD 




3D 


4D 


1 1 


1 1 


e. Social studies 


iD 


oD 


iD 




3D 


4D 


1 • 


1 


2. Spring 

a. Reading 


iD 


oD 


iD 




3D 


4D 


3rd Qtr 


4th Qtr 


b. English/Language 


iD 


oD 


iD 




3D 


4D 


1 1 


1 1 


c. Math 


iD 


oD 


iD 




3D 


4D 


1 1 




d. Science 


iD 


oD 


iD 




3D 


4D 


1 1 


1 1 


e. Social studies 


iD 


oD 


iD 




3D 


4D 


1 * 


• 
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Obs Initial Date School Teacher Time to 

Expository Reading Comprehension Ciassroom Observation Form 



Background Information (or label) 

Observer 

School 

District 

Teacher 

State 



Grade 



Today's Date / / 

mm dd yyyy 



Start time 


a.m. 


p.m. 


End time 


a.m. 


p.m. 



Subject (circle all that apply): 

Social Studies Science 

Reading/LA Intervention 



Maximum number of students 
observed in classroom 




Maximum number of adults observed 
providing instruction or educational 
support in the classroom (including 
teacher) 



Number 



Any special circumstances that interrupted instruction? (please explain) 



If this is an intervention observation please check below: 



ProjectCRISS Read About (Scholastic) Reading for Knowledge (SFA) 

Read for Real SRA 



Note to Rater: Focus on primary teacher for rating purposes. If student teacher is leading class, please do not 
observe. 
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Obs Initial Date School Teacher Time to 

Part 1. 1®' Interval (Start time: End Time: ) 



Comprehension 



Before Reading 


Models 


Explains, 

Reviews, 

Names 


Student 
P ractice 


Notes 


1. The teacher/student activates prior 
knowledge and/or previews text before 
reading (e.g., shares background 
information about the title, author, content; 
reviews relevant content from previous 
lessons: picture walk; makes predictions: 
makes connections) 










Before, During, or After Reading 


Models 


Explains, 

Reviews, 

Names 


Student 
P ractice 




2. Instruction using text features (sub-heads, 
captions, charts, maps, graphs, sidebars, 
bold and italicized words) to interpret text 








r 


3. Using text structure to teach/identify 
compare-contrast, cause-effect, or 
problem-solution (may include story 
grammar/elements if using informational 
text that has a narrative structure) 








4a. Explicit comprehension instruction that 
teaches students how to use strategies, 
such as main idea, summarizing, drawing 
conclusions, visualizing events, evaluatincy 
predictions, identifying fact vs. opinion, H 
sequencing, monitoring for 1 

comprehension, other | 






if 


4b. Generating questions ^ 








5. Asks students to justify or elaborate their 
responses (e.g., teacher asks "why," "how 
did you reach that conclusion," etc.) 






During or After Reading 


Models 


Explains, 

Reviews, 

Names 


Student 

Practice 




6. T eacher asks questions based on material 

in the text that require one of the following: 
(making inferences [Ml], 
summarizing/finding main ideas [S], 
drawing conclusions [DC] or some other 
complex skill). Please indicate in notes 
your best guess at type of skill (e.g.. Ml, 
S, DC, other) 










7. T eacher elaborates, clarifies, or links 
concepts during text reading. May be an 
elaboration of student responses. 
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Obs Initial Date School Teacher Time to 



Vocabulary (includes concepts, terminology, ideas; may be technical or complex content-area vocabulary) 





Tally 


Notes 


1. The teacher provides an explanation, a definition, or an 
example. 






2. The teacher elaborates or extends a definition. May include 
using multiple or contrasting examples to pinpoint a definition: 
further developing or paraphrasing the definition by 
incorporating ideas from students' responses, examples, and 
experiences: or discussing multiple-meanings. 




3. The teacher uses visuals, gestures related to word meaning, 
facial expressions, pictures, or demonstrations to 
discuss/demonstrate word meanings. 




4. T eaches word-learning strategies- using context clues, word 
parts, or root meaning. 




5. Asks students to do something that requires knowledge of 
words (e.g., answer questions, define words, make 
sentences, find words based on clues, physically demonstrate 
meaning). 




6. G Ives students opportunity to apply word-learning strategies- 
using context clues, word parts, root meaning. 





Grouping arrangements and text reading (Code during each 1 5-minute cycle) 



Teacher is workina with 

(Choose all that apply.) 


Grouoina arranaements 

(Choose all that apply.) 




Text Readina 

(Choose all that apply.) 


1 . Whole class (>75% of 
class) 

2. Large group (> 6 students, 

< 75% of class) 

3. Small groups (3-6 students) 

4. Pairs 

5. An Individual 

6. No direct student contact 


1 . Whole class (>75% of class) 

2. Large group (> 6 students, < 75% of 
class) 

3. Small groups (3-6 students) 

4. Pairs 

5. Working with an adult (other than a 
teacher) 

6. Reading individually 

7. Doing individual work (e.g., seat 
work/computer) 


1 . Supported oral reading (includes 
choral reading) 

2. Independent silent reading 

3. Independent oral reading 

4. Teacher reads aloud 

5. Teacher reads aloud with 
students following along silently 

6. Text not used for comprehension 
instruction 


1 2 3 4 5 6 


1 2 3 4 5 6 7 


1 2 3 4 5 6 



Materials (Check materials that were used during the interval): 



1. ___ maps, charts, and graphs (Including 

workbook, worksheet pages, or graphic 
organizers) 

2. other visuals (with or without print) 

3. text-basal 

4. text-trade book, authentic text (e.g., poem, 

nonfiction book, song) 

5. textbook-science 

6. textbook-social studies 

7. text handouts from supplementary materials, 

magazines 



8. text-student made 

9. computers 

10. audio-tapes (e.g., books on tape) 

11. workbook pages and worksheets 

12. chalk board or equivalent (dry erase, easel, 

overhead) 

13. videos 

14. paperand pencil 

15. other 
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Obs Initial Date School Teacher Time to 

Draft 08/05 

Part II. 

Answer the following questions at the end of your observation: 



Features of Effective Instruction 
During/After instruction, the teacher: 



1. G ave inaccurate and/or confusing explanations or feedback. 


Y N 




2. M issed opportunity to correct or address error. 


Y N 




3. P rovided opportunities for most students to participate actively 
during teacher-led instruction. 


Y N 




4. Paced instruction so that the length of the comprehension or 
vocabulary activities were appropriate for this age group. 


Y N 




5. Teaches using outlining and note taking. 


Y N 




6. Uses graphic organizers (e.g., KWL, Venn diagrams). 


Y N 




7. Keeps students thinking for 2-F seconds before calling on a 
student to respond to complex questions. 


Y N 




8. G ives independent/pairs/small-group practice in answering 
comprehension questions or applying comprehension 
strategy(ies) with expected product. (Can include response 
journals if a comprehension strategy is entailed.) 


Y N 




9. Uses writing activities in response to reading (does not include 
fill in the blank or one word answers). 


Y N 





Based on your overall ob servations, rate the quality of the comprehension instruction you observed. 





Not Observed 


Minimal/Erratic 


Partially 

Effective 


Good 


Excellent 


10. Comprehension 


N/0 


1 


2 


3 


4 



Based on your overall observations, rate the teachers’ management/responsiveness to stud 


lents*. 




Minimal/Poor 


Fair 


Good 


Excellent 


11. The teacher maximized the amount of time available for instruction. 


1 


2 


3 


4 


12. The teacher managed student behavior effectively to avoid 
disruptions and provide productive learning environments. 


1 


2 


3 


4 


13. The teacher redirected discussion if a student response was 
leading the group off topic/focus. 


N/0 ^ 


2 


3 


4 



* Items are adapted from Teacher Competency Checklist (Foorman & Schatschneider, 2003). 



Based on your overall observations, rate si 


ludent engagement during the observation. 


Few engaged 


Many engaged 


Most engaged 


14. Student engagement during the first half of 
the observation session. 


1 


2 


3 


15. Student engagement during the remainder 
of the observation session. 


1 


2 


3 
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Intervention-Specific Classroom Observation Form: CRISS 



Background Information (or label) 



Observer 


Todav’s Date / 


/ 


School 


mm dd 


yyyy 


District 


Start time 


a.m. 


Teacher 


End time 


a.m. 


State 



Grade 



Maximum number of 
students observed in 
classroom 



Subject (circle all that apply): 

Social Studies Science 




Reading/LA 



Intervention 




Number 



INMximum number of adults 



onserved providing instruction or 
educational support in the 
classroom (including teacher) 



Any special circumstances that interrupted instruction? (please explain) 



Notes to Rater: 

1. Focus on regular classroom teacher for rating purposes. If student teacher or substitute is leading 
class, please do not observe. 

2. Make sure that the teacher is teaching with expository text for your observation . 
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Star each section that you observe today. Answer the questions in that section 
only. Do not answer the questions in the sections that you do not observe. 



Does the teacher... 




About how manv 
of the students... 








Section I. Preparing for 
Understanding 












1. Provide instruction or lead 
activities to generate 
background knowledge about 
(or review) a topic or concept 
before students read about it? 


Y N 


1. Actively participate 
in the activities or 
discussion? 


Few 


Many 


Most 


2. Help students set goals and 
determine a purpose before the 
students begin reading? 


Y N 










Section II. Engaging Students with 
Content and Transforming 
Information 












3. Have students read a written 
text? 


Y N 


3. Actively participate 
in reading a written 
text? 


Few 


Many 


Most 


4a. Lead students during and/or 
after reading in transforming^ 
information activities (e.g., 
graphic organizer, guided 1 I 
discussion)? 


t 


4a. Actively 
participate in 
trmisforming 
■formation 
activities? 


Few 


Many 


Most 


4b. Include informal or formal 
writing in the transforming 
activities? 


Y N 


4b. Actively 
participate in 
informal or formal 
writing? 


Few 


Many 


Most 


5. Use the transforming activities 
to teach the content of the 
lesson? 


Y N 










6. Discuss or reflect on students’ 
metacognitive processes during 
the transforming activities? 


Y N 


6. Actively participate 
in metacognitive 
discussion or 
reflection? 


Few 


Many 


Most 


Section III. Reflecting on Content 
and Learning Processes 












7. Lead the whole class in a 

reflection discussion at the end 
of the lesson using questions 
such as: 

A) Metacognition: How did you 
evaluate your comprehension? 


Y N 


7. Actively participate 
in the reflection 
discussion at the end 
of the lesson? 


Few 


Many 


Most 
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B) Background knowledge: Did I 
assist you in thinking about what 
you already knew? 

C) Purpose Setting: Did you have 
clear purposes? 

D) Active Involvement: How were 
you actively engaged? 

E) Discussion: How did discussion 
clarify your thinking? 

F) Writing: How did you use 
writing to help you learn? 

G) Transformation: What were the 
different ways you transformed 
information? How did this help 
you? 

H) Teacher modeling: Did I do 
enough modeling? 



Please note: You may see all three sections in one sitting. Or you may see Sections I and II, or 
Sections II and III, or Section II alone. You should never see Sections I and III together. It is also 
unlikely that you will see Sections I alone or Section III alone. 

P 
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Intervention- Specific Classroom Observation Form: ReadAbout 



Background Information (or label) 

Observer 

School 
District 
Teacher 
State 

Grade 



Number 

Maximum number of adults 
observed providing instruction or 
educational support in the 
classroom (including teacher) 

Any special circumstances that interrupted instruction? (please explain) 



Number 

Maximum number of 
students observed in 
classroom 
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Check which of the following were observed: 

o Computer-based instruction (Number of students working on the computer: ; 

Time [e.g., 18 minutes]) 

o Teacher-ied smaii group instruction (Number of students in group: ; Time ) 

o independent work by students (Number of students doing independent work: ; 

Time ) 



A. Answer these questions while observing the lesson. 



Computer-based instruction 


Teacher-led small group: 
Did the teacher 


ludepeudeut work: Were 
the studeuts 


1. Was the computer 
program implemented 
for the required time? 


Y N 


1. Explain and model 
the strategy or skill? 

^ 1 


Y N 

0 * 


\h Using the 

■rogram 

Imaterials? 


Y N 


2. Were students 
engaged in the 
computer program’s 
activities? 


Y N 


2. Provide A I 

opportunities 1 

guided pp^ic^^ | 


^ N 


I2. Actively 
lengaged in 
completing 
the 

assignment? 


Y N 


3. Provide students 
instruction on the 
selected 6+1 Writing 
Trait? 


Y N 


3. What was the primary focus 
of the students’ independent 
activities? 

0 Author’s purpose 
0 Main idea/details 
0 Draw conclusions 
0 Fact/opinion 
0 Text structure 
(cause/effect, 
compare/contrast, 
sequence of events, 
problem/solution) 

0 Make inferences 
0 Summarizing 
0 Visualizing 
0 Setting purpose 
0 Monitoring (including 
rereading and 
repairing) 

0 Questioning 


4. Use the program 
materials? 


Y N 


5. Which components of 
the computer module 
was the student 
working on (circle all 
that apply and were 
observed)? 

1. Initial explanation 
and modeling of 
the strategy 

2. Guided practice 

3. Writing prompt 

4. Review of key 
vocabulary 

5. Skill tutorial 


Y N 

Y N 

Y N 

Y N 

Y N 


5. What was the primary focus 
of the teacher-led instruction? 

o Author’s purpose 
o Main idea/details 
o Draw conclusions 
o Fact/opinion 
o Text structure 
(cause/effect, 
compare/contrast, 
sequence of events, 
problem/solution) 
o Make inferences 
o Summarizing 
o Visualizing 
o Setting purpose 
o Monitoring (including 
rereading and 
repairing) 
o Questioning 
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B. Rate the following items based on the overall observation. 





Most of the 
time 


Some of the 
Time 


Rarely 


Not at all 


Not 

observed 


1. Were the students actively 
engaged in instruction? 












2. Did the teacher monitor on- 
going student 
progress/work during the 
lesson? 












3. Did the teacher provide 
corrective feedback to 
students? 




1 
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Intervention- Specific Classroom Observation Form: Reading For Knowledge 

Days 2 and 4 version 



Background Information (or label) 



Observer 

School 

District 

Teacher 

State 



Grade 



Today’s Date / / 

mm dd yyyy 




Reading/LA 



a.m. p.m. 
a.m. p.m. 



Science 

Intervention 



Number 

Maximum number of 
students observed in 
classroom 



Number 

Maximum number of adults 
observed providing instruction or 
educational support in the 
classroom (including teacher) 



Any special circumstances that interrupted instruction? (please explain) 



Please record the following: 

l.Unit# 2. Week# 3. Day# 4. Book Title 



Notes to Rater: 

1. Focus on the regular classroom teacher for rating purposes. If student teacher or substitute teacher 
is leading class, please do not observe. 

2. If today’s class period includes testing, please do not observe and reschedule the observation visit. 
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A. Answer these questions while observing the lesson. 



To what extent does the 
teacher follow the 
recommended procedures 
suggested in the teachers’ 
manual... 








About how many of 
the students... 








I. Set the Stage 










Few 

< 

25% 


Many 

25%- 

75% 


Most 

> 

75% 


a. Post the reading goal? 


Little 


Some 


Most 










h. Present the reading goal? 


Little 


Some 


Most 


b. Actively engage in 
listening to the 
presentation of the 
reading goal? 


Few 


Many 


Most 


c. Present the cooperative 
learning goal? 


pi 


fSome 

1 


Most 


c. Actively engage in 
listening to the 
presentation of the 
cooperative learning 
goal? 


Few 


Many 


Most 


d. Provide the vocabulary 
instruction or practice? 


Little 


Some 


Most 


d. Actively participate 
in whole group 
vocabulary instruction 
and practice? 


Few 


Many 


Most 


II. Active Instruction 










Few 

< 

25% 


Many 

25%- 

75% 


Most 

> 

75% 


a. Use a whole group or 
partner activity to discuss key 
points about the day’s 
skill/strategy? 


Little 


Some 


Most 


a. Actively participate 
in the whole group or 
partner activity to 
discuss key points 
about the day’s 
skill/strategy? 


Few 


Many 


Most 


b. Provide feedback and 
prompts to partner pairs 
during partner reading? 


Little 


Some 


Most 


b. Actively participate 
in partner reading and 
discussion? 


Few 


Many 


Most 


c. Chart individual students’ 
progress on the setting goals 
and charting progress forms 
during partner reading? 


Little 


Some 


Most 










d. Review routines for Team 
Talk discussion? 


Little 


Some 


Most 










e. Read aloud Team Talk 
questions? 


Little 


Some 


Most 










f. Circulate the classroom and 
monitor team discussions and 


Little 


Some 


Most 


f. Actively participate 
in the group Team 


Few 


Many 


Most 
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provide prompts? 








Talk discussion? 








g. Ask team members to share 
with the class their responses 
and reasoning to Team Talk 
questions? 


Little 


Some 


Most 


g. Appear engaged in 
the whole group Team 
Talk discussion? 


Few 


Many 


Most 



B. Answer these two overall questions at the end of the lesson. 



The teacher followed the recommended pacing for the lesson. 
(Recommended pacing is 35 minutes +/- 5 minutes.) 






Y N 


The teacher awarded cooperation and/or improvement points at 
lesson. ^ 


some point in the 


Y N 
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Intervention-Specific Classroom Observation Form: Read for Real 

Phase: Learn 



Background Information (or label) 



Observer 




Today’s Date / / 




School 




mm dd yyyy 


District 




Start time 


a.m. p.m. 


Teacher 




End time 


a.m. p.m. 


State 






Grade 




S^ect (circle all that apply): 
Jk Social Studies 


Science 




n 


R 


■Jh ^Reading/LA 


Intervention 




U 


P 



Number 

Maximum number of 
students observed in 
classroom 



Number 

Maximum number of adults 
observed providing instruction or 
educational support in the 
classroom (including teacher) 



Any special circumstances that interrupted instruction? (please explain) 
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Phase: Learn 



Does the teacher follow the procedures suggested in the Teacher Guide? 

1. Unit Introduction 
The teacher: 

a. Asks a student to read the title of the unit. Y N 

b. Comments/asks questions to pique interest. Y N 

c. Introduces the reading partner for the unit. Y N 



2. Before Reading 
The teacher: 

a. Asks a student to read the explanation of the Before Reading focus strategy. Y N 

b. Discusses the Before Reading focus strategy with the students. Y N 

c. Encourages students to apply the Before Reading focus strategy. Y N 

d. Calls students’ attention to the “My Thinking” box. Y N 

e. Asks a student to read what the reading partner says about the Before Reading focus 



strategy by reading the information ia 



Student Engagement 

About how many students 
Few Man 




ation imte “mJ^i 

engaged OTrin" this 
^ Most 



M^^inking” 

I 



box. Y N 



portion of the lesson? 



3. During Reading 
The teacher: 

a. Asks a student to read the explanation of the During Reading focus strategy. Y N 

b. Discusses the During Reading focus strategy with the students. Y N 

c. Asks a student to read the information in the “My Thinking” box. Y N 

d. Encourages students to share their thinking about the During Reading focus strategy. 
YN 

e. Reads or asks students to read the selection aloud. 

Never Sometimes Always 

f. Stops and discusses the “My Thinking” notes at each “red strategy button.” 

Never Sometimes Always 

g. Asks comprehension questions. 

Never Sometimes Always 



Student Engagement 

About how many students are engaged during this portion of the lesson? 
Few Many Most 
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4. After Reading 
The teacher: 

a. Asks students to read the After Reading foeus strategy. Y N 

b. Diseusses the After Reading foeus strategy with the students. Y N 

c. Asks a student to read the information in the “My Thinking” box. Y N 

d. Calls on students to implement the After Reading foeus strategy. Y N 

Comprehension 

e. Administers the comprehension test Y N 

f. Corrects tests with the class. Y N 

g. Discusses responses. Y N 

Organizing Information 

h. Asks a student to read the information from the reading partner. Y N 

i. Conducts activity with graphic organizer. Y N 

Writing for Comprehension 

j. Asks a student to read the information from the reading partner Y N 

k. Reads or asks a student to read the summary. Y N 

l. Identifies how the paragraphs and sentences in the summary correspond to the 
information on the graphic organizer. Y N 

m. Discusses the three parts of a summary: 



Introduction 


YN 


Body 


YN 


Conclusion 


YN 



n. Informs students that the author went through several steps (rough drafts, editing, and 
proofreading) to get to finished product. Y N 

Vocabulary 

o. Instructs students in the vocabulary skill. Y N 

p. Asks students to complete the vocabulary activity: 

as a whole class in small groups independently in partners 



Fluency 

q. Asks a student to read the fluency tip. Y N 

r. Asks a student to read the selection. Y N 

s. Gives students time to practice the selection. Y N 

Student Engagement 

About how many students are engaged during this portion of the lesson? 
Few Many Most 



English Learners 

Integrates activities for English Language Learners into the lesson. 
Never Sometimes Always 
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Intervention-Specific Classroom Observation Form: Read for Real 

Phase: Practice 



Background Information (or label) 



Observer 

School 

District 

Teacher 

State 



Grade 



Maximum number of 
students observed in 
classroom 



Today’s Date / / 

mm dd yyyy 



Start time 


a.m. 


p.m. 


End time 


a.m. 


p.m. 



Subject (circle all that apply): 



u n*' 



Social Studies 



Keading/LA 



Number 



Science 

Intervention 



Number 



Maximum number of adults 
observed providing instruction or 
educational support in the 
classroom (including teacher) 



Any special circumstances that interrupted instruction? (please explain) 
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Phase: Practice 



Does the teacher follow the procedures suggested in the Teacher Guide... 

1. Before Reading 
The teacher: 

a. Tells student there is a new reading partner for this story. Y N 

b. Tells students the reading partner will give them suggestions at the beginning and end 
of the selection to help them practice the strategies. Y N 

c. Asks a student to read the Before Reading focus strategy. Y N 

d. Discusses the Before Reading focus strategy with the students. Y N 

e. Asks students to implement the Before Reading focus strategy. Y N 

f. Lists students’ comments on board. Y N 



Student Engagement 

About how many students are engaged during this po: 



_Many 



2. During Reading ■ 

The teacher: 

a. Asks a student to read the During Reading focus strategy. Y N 

b. Asks a student to read the note from the reading partner. Y N 

c. Reminds students to write notes about the During Reading focus strategy the way the 
reading partner did in the previous selection. Y N 

d. Reads or asks a student to read the first paragraph or two of the selection. Y N 

e. Discusses selection with students. Y N 



lesson? 



f. Remainder of the selection is read: (Check all that apply.) 

as a whole class in small groups in partners independently. 

g. Reminds students to stop at the red buttons and write notes on their paper. Y N 

h. Asks comprehension questions. 

Never Sometimes Always 



Student Engagement 

About how many students are engaged during this portion of the lesson? 
Few Many Most 
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3. After Reading 
The teacher: 

a. Asks students to read the After Reading foeus strategy. Y N 

b. Diseusses or asks question about the After Reading focus strategy. Y N 

c. Gives written assignment highlighting the After Reading focus strategy. Y N 

Comprehension 

e. Administers the comprehension test. Y N 

f. Corrects tests with the class. Y N 

g. Discusses responses. Y N 

Organizing Information 

h. Asks students to complete graphic organizer. 

Writing for Comprehension 

i. Asks students to write a summary based on their completed graphic organizer. Y N 

Vocabulary 

j. Instructs students in the vocabulary skill. Y N 

k. Asks students to complete the vocabulary activity: 

as a whole class in small groups independently in partners 

Fluency 

l. Asks a student to read the fluency tip. Y N 

m. Asks a student to read the selection. Y N 

n. Gives students time to practice the selection. Y N 

Student Engagement 

About how many students are engaged during this portion of the lesson? 

Few Many Most 

English Learners 

Integrates activities for English Language Learners into the lesson. 

Never Sometimes Always 
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Intervention-Specific Classroom Observation Form: SRA 



Background Information (or label) 



Observer 

School 

District 

Teacher 

State 



Today’s Date / / 





mm dd 


yyyy 


Start time 




_ a.m. 


End time 


p.m. 


a.m. 




p.m. 





Grade 



Subject (circle all that apply): 

Social Studies Science 



Reading/LA Intervention 



Maximum number of 
students observed in 
classroom 





laximum number of adults 
observed providing instruction or 
educational support in the 
classroom (including teacher) 



Number 



Any special circumstances that interrupted instruction? (please explain) 
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Day 1: Social Studies/Science 


Part I. Lesson Segments 


BEFORE READING 


Does the teacher. . . 


1. Discuss students ’ prior experiences and knowledge by helping 
them make connections with the content of the lesson 

(i.e., text to self, text to world, or text to text? Y N 


2. Ask a volunteer to read the title aloud? Y N 


3. Ask a volunteer to identify the skill that will be practiced in the lesson ? Y N 


4. Direct students’ attention to the text? Y N 


5. Discuss the characteristics of the text with students, such as 
highlighted or boldfaced words, boxed text, subheads 

visuals, captions, etc. ? Y N 


6. Ask students to demonstrate understanding of the lesson’s concepts and 

/or vocabulary (i.e., examples, synonyms, locate, point to)? Y N 


DURING/AFTER READING 


Does the teacher. . . 


1. Read each section of the text aloud? Y N 


2. Ask the students to follow along as he/she reads each section of the text? Y N 


3. Involve students in the discussion of how the features/organization of the text is used as a 
means of drawing the reader into the selection (e.g., graphic organizers, maps, visuals, 
models, numbers, subheads, steps, etc.)? Y N 


4. Give students practice using any one or more of the following strategies? Y N 

(check all that apply) 

Predicting/confirming predictions 
Summarizing 

Asking questions/finding answers 

Monitoring/clarifving ( includes helping students use strategies for dealing with 
unfamiliar words in the text) 

Visualizing 

Monitoring for meaning (adiusting speed and/or rereading) 

Making connections 


Part II. Other Components 


The teacher also. . . 

(circle one or more) 

1. has volunteers look up the words in the dictionary for more detailed 
definitions of vocabulary 

2. models comprehension strategies when students struggle with practicing how to use 
comprehension strategy 

3. provides guided practice with map skills 
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Day 2; Social Studies/Science 



Part I. Lesson Segments 


BEFORE READING 


Does the teacher. . . 


1. Ask a volunteer to read the title aloud? 


Y 


N 


2. Remind students that the text on this page relates to the previous page? 


Y 


N 


3. Direct students’ attention to the text? 


Y 


N 


4. Discuss with students the characteristics of the text, such as highlighted 






or boldface words, boxed text, subheads, visuals, captions, etc. ? 


Y 


N 


DURING/AFTER READING: 


Does the teacher. . . 


1. Read each section of the text aloud? 


Y 


N 


2. Ask the students to follow along as he/she reads each section of the text? 


Y 


N 


3. Involve students in the discussion of how the features/organization of the 
text is used as a means of drawing the reader into the selection 






(e.g., graphic organizers, maps, visuals, models, numbers, subheads, steps, etc.)? 


Y 


N 



4. Give students practice using any one or more of the following strategies? 
(check all that apply) 

Predicting/confirming predictions 

Summarizing 



_Asking questions/finding answers 
JAonitoring/clarifying ( including helping students use 
strategies for dealin^mth unfamiliar words in the text) 
fVisualizing ^ 

Jdonitoring for meam^^ (adjusting speed and/or rereading) 
JAaking connections ^ 



Part II. Other Components 



The teacher also. . . 

(circle one or more) 

1. models comprehension strategies when students struggle with 
practicing how to use a comprehension strategy 

2. provides guided practice with map skills, charts, graphs, etc. 

3. provides modeling for other skill work such as taking notes, drawing conclusions, etc. 

4. uses a “think aloud” technique to model a comprehension strategy 

5. invites students to “think aloud” 
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Day 3: Science 


Part I. Lesson Segments 


DURING/AFTER READING: 


Does the teacher. . . 


1. Read each section of the text aloud? 




Y 


N 


2. Ask the students to follow along as he/she reads each section of the text? 


Y 


N 


3. Have students analyze the features/organization of the text in terms of how 
it is used as a means of drawing the reader into the selection (e.g., graphic 
organizers, maps, visuals, models, numbers, subheads, steps, etc.)? 


Y 


N 


4. Give students practice using any one or more of the following strategies? 
(check all that apply) 

Predicting/confirming predictions 
Summarizing 

Asking questions/finding answers 
Monitoring/clarifving ( including helping students use 
strategies for dealing with unfamiliar words in the text) 
Visualizing 

Monitoring for meaning (adiusting speed and/or rereading) 
Making connections 


Y 


N 


Part II. Other Components 




The teacher also. . . 

(circle one or more^|^ 


0 A 


n 






1. models comprehension strategies when students struggle with 
practicing how to use a comprehension strategy 

2. provides guided practice with map skills 
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Day 3: Social Studies 


Part I. Lesson Segments 


BEFORE READING 


Does the teacher. . . 


1. Direct students’ attention to the text? Y N 


2. Discuss with students the characteristics of the text such as 

highlighted or boldfaced words, boxed text, subheads, visuals, captions, etc. ? Y N 


DURING/AFTER READING: 


Does the teacher. . . 


1. Read each section of the text aloud? Y N 


2. Ask the students to follow along as he/she reads each section of the text? Y N 


3. Have students analyze the features/organization of the text in terms of 
how it is used as a means of drawing the reader into the selection 

(e.g., graphic organizers, maps, visuals, models, numbers, subheads, steps, etc.)? Y N 


4. Give students practice using any one or more of the following strategies Y N 

(check all that apply) 

Predicting/confirming predictions 
Summarizing 

Asking questions/finding answers 
Monitoring/clarifving ( including helping students use 
strategies for dealing with unfamiliar words in the text) 

Visualizing 

Monitoring for meaning (adiusting speed and/or rereading) 

Making connections 


Part II. Other Components 


The teacher also. . . 

(circle one or more) 

1. models comprehension strategies when students struggle with 
practicing how to use a comprehension strategy 

2. provides guided practice with map skills 



DRAFT 



C.51 





